[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260430 07:56:58.894744 8670 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.119.190:35737
I20260430 07:56:58.896760 8670 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260430 07:56:58.898202 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:56:58.927482 8679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:56:58.927603 8677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:56:58.928313 8676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:56:58.930763 8670 server_base.cc:1061] running on GCE node
I20260430 07:57:00.467098 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:57:00.467383 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:57:00.467617 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535820467587 us; error 0 us; skew 500 ppm
I20260430 07:57:00.468457 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:57:00.475311 8670 webserver.cc:492] Webserver started at http://127.8.119.190:41445/ using document root <none> and password file <none>
I20260430 07:57:00.476583 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:57:00.476847 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:57:00.477358 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:57:00.484505 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/master-0-root/instance:
uuid: "0ac3f16b3d5b4753b196ac832dad0046"
format_stamp: "Formatted at 2026-04-30 07:57:00 on dist-test-slave-6ct4"
I20260430 07:57:00.494613 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260430 07:57:00.501554 8685 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:00.503153 8670 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260430 07:57:00.503635 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/master-0-root
uuid: "0ac3f16b3d5b4753b196ac832dad0046"
format_stamp: "Formatted at 2026-04-30 07:57:00 on dist-test-slave-6ct4"
I20260430 07:57:00.504825 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:57:00.608199 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:57:00.609988 8670 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260430 07:57:00.610531 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:57:00.703860 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.190:35737
I20260430 07:57:00.703964 8736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.190:35737 every 8 connection(s)
I20260430 07:57:00.714740 8737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:00.742813 8737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046: Bootstrap starting.
I20260430 07:57:00.750669 8737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:00.752884 8737 log.cc:826] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046: Log is configured to *not* fsync() on all Append() calls
I20260430 07:57:00.758652 8737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046: No bootstrap required, opened a new log
I20260430 07:57:00.783502 8737 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac3f16b3d5b4753b196ac832dad0046" member_type: VOTER }
I20260430 07:57:00.784164 8737 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:00.784482 8737 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac3f16b3d5b4753b196ac832dad0046, State: Initialized, Role: FOLLOWER
I20260430 07:57:00.787318 8737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac3f16b3d5b4753b196ac832dad0046" member_type: VOTER }
I20260430 07:57:00.789372 8737 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 07:57:00.789791 8737 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 07:57:00.790190 8737 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:00.794952 8737 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac3f16b3d5b4753b196ac832dad0046" member_type: VOTER }
I20260430 07:57:00.795876 8737 leader_election.cc:304] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac3f16b3d5b4753b196ac832dad0046; no voters:
I20260430 07:57:00.797940 8737 leader_election.cc:290] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 07:57:00.798529 8740 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:00.804831 8740 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac3f16b3d5b4753b196ac832dad0046, State: Running, Role: LEADER
I20260430 07:57:00.806012 8740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac3f16b3d5b4753b196ac832dad0046" member_type: VOTER }
I20260430 07:57:00.806772 8737 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 07:57:00.827690 8742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac3f16b3d5b4753b196ac832dad0046. Latest consensus state: current_term: 1 leader_uuid: "0ac3f16b3d5b4753b196ac832dad0046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac3f16b3d5b4753b196ac832dad0046" member_type: VOTER } }
I20260430 07:57:00.827629 8741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ac3f16b3d5b4753b196ac832dad0046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac3f16b3d5b4753b196ac832dad0046" member_type: VOTER } }
I20260430 07:57:00.828478 8742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [sys.catalog]: This master's current role is: LEADER
I20260430 07:57:00.828531 8741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [sys.catalog]: This master's current role is: LEADER
I20260430 07:57:00.836741 8750 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 07:57:00.851193 8750 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 07:57:00.853731 8670 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260430 07:57:00.874611 8750 catalog_manager.cc:1357] Generated new cluster ID: 12751d31068d44a3824057caab46b4b9
I20260430 07:57:00.875733 8750 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 07:57:00.907639 8750 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 07:57:00.910424 8750 catalog_manager.cc:1514] Loading token signing keys...
I20260430 07:57:00.936236 8750 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046: Generated new TSK 0
I20260430 07:57:00.937587 8750 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 07:57:00.987375 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:57:01.000113 8759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:01.003129 8760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:01.008672 8762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:57:01.009418 8670 server_base.cc:1061] running on GCE node
I20260430 07:57:01.010636 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:57:01.011029 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:57:01.011234 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535821011215 us; error 0 us; skew 500 ppm
I20260430 07:57:01.012297 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:57:01.018493 8670 webserver.cc:492] Webserver started at http://127.8.119.129:34713/ using document root <none> and password file <none>
I20260430 07:57:01.019346 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:57:01.019716 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:57:01.020098 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:57:01.021802 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-0-root/instance:
uuid: "916b9fc1bae040e681e4b82126f3ecad"
format_stamp: "Formatted at 2026-04-30 07:57:01 on dist-test-slave-6ct4"
I20260430 07:57:01.031307 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260430 07:57:01.039112 8767 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:01.040503 8670 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260430 07:57:01.041150 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-0-root
uuid: "916b9fc1bae040e681e4b82126f3ecad"
format_stamp: "Formatted at 2026-04-30 07:57:01 on dist-test-slave-6ct4"
I20260430 07:57:01.042421 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:57:01.108843 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:57:01.110936 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:57:01.113521 8670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 07:57:01.118372 8670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 07:57:01.118690 8670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:01.119186 8670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 07:57:01.119496 8670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:01.256398 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.129:42251
I20260430 07:57:01.256747 8829 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.129:42251 every 8 connection(s)
I20260430 07:57:01.266575 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:57:01.282689 8834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:01.286417 8835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:01.293025 8837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:57:01.302307 8670 server_base.cc:1061] running on GCE node
I20260430 07:57:01.303627 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:57:01.303855 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:57:01.304203 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535821304173 us; error 0 us; skew 500 ppm
I20260430 07:57:01.305047 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:57:01.310190 8670 webserver.cc:492] Webserver started at http://127.8.119.130:33907/ using document root <none> and password file <none>
I20260430 07:57:01.310812 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:57:01.311064 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:57:01.311379 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:57:01.312474 8830 heartbeater.cc:344] Connected to a master server at 127.8.119.190:35737
I20260430 07:57:01.312997 8830 heartbeater.cc:461] Registering TS with master...
I20260430 07:57:01.313078 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-1-root/instance:
uuid: "35eb09ea173d4264b201b528036d5b52"
format_stamp: "Formatted at 2026-04-30 07:57:01 on dist-test-slave-6ct4"
I20260430 07:57:01.314349 8830 heartbeater.cc:507] Master 127.8.119.190:35737 requested a full tablet report, sending...
I20260430 07:57:01.320111 8702 ts_manager.cc:194] Registered new tserver with Master: 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251)
I20260430 07:57:01.322865 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260430 07:57:01.325347 8702 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51922
I20260430 07:57:01.328348 8842 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:01.329913 8670 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260430 07:57:01.330655 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-1-root
uuid: "35eb09ea173d4264b201b528036d5b52"
format_stamp: "Formatted at 2026-04-30 07:57:01 on dist-test-slave-6ct4"
I20260430 07:57:01.330998 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:57:01.356014 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:57:01.357999 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:57:01.360733 8670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 07:57:01.364694 8670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 07:57:01.365001 8670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:01.365343 8670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 07:57:01.365568 8670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:01.543455 8904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.130:39379 every 8 connection(s)
I20260430 07:57:01.543857 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.130:39379
I20260430 07:57:01.553678 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:57:01.567170 8909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:01.570395 8908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:57:01.573640 8905 heartbeater.cc:344] Connected to a master server at 127.8.119.190:35737
W20260430 07:57:01.573909 8911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:57:01.574137 8905 heartbeater.cc:461] Registering TS with master...
I20260430 07:57:01.575198 8905 heartbeater.cc:507] Master 127.8.119.190:35737 requested a full tablet report, sending...
I20260430 07:57:01.577047 8670 server_base.cc:1061] running on GCE node
I20260430 07:57:01.577749 8702 ts_manager.cc:194] Registered new tserver with Master: 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:01.578004 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:57:01.578308 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:57:01.578500 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535821578482 us; error 0 us; skew 500 ppm
I20260430 07:57:01.579070 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:57:01.579577 8702 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51924
I20260430 07:57:01.583758 8670 webserver.cc:492] Webserver started at http://127.8.119.131:33839/ using document root <none> and password file <none>
I20260430 07:57:01.584378 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:57:01.584616 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:57:01.585022 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:57:01.586254 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-2-root/instance:
uuid: "ec08915611e344558a04e8a9c11b6b0a"
format_stamp: "Formatted at 2026-04-30 07:57:01 on dist-test-slave-6ct4"
I20260430 07:57:01.591424 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260430 07:57:01.596701 8916 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:01.597935 8670 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260430 07:57:01.598399 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-2-root
uuid: "ec08915611e344558a04e8a9c11b6b0a"
format_stamp: "Formatted at 2026-04-30 07:57:01 on dist-test-slave-6ct4"
I20260430 07:57:01.598774 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1777535818658779-8670-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:57:01.620235 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:57:01.622226 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:57:01.625240 8670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 07:57:01.631825 8670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 07:57:01.632053 8670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:01.633109 8670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 07:57:01.633754 8670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.001s user 0.002s sys 0.000s
I20260430 07:57:01.789009 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.131:42455
I20260430 07:57:01.789157 8978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.131:42455 every 8 connection(s)
I20260430 07:57:01.817981 8979 heartbeater.cc:344] Connected to a master server at 127.8.119.190:35737
I20260430 07:57:01.818560 8979 heartbeater.cc:461] Registering TS with master...
I20260430 07:57:01.820127 8979 heartbeater.cc:507] Master 127.8.119.190:35737 requested a full tablet report, sending...
I20260430 07:57:01.825736 8702 ts_manager.cc:194] Registered new tserver with Master: ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:01.825904 8670 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.029412704s
I20260430 07:57:01.828397 8702 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51938
I20260430 07:57:01.897521 8702 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51952:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 07:57:01.901973 8702 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 07:57:02.082389 8793 tablet_service.cc:1511] Processing CreateTablet for tablet 8a8897623436482eb062dc3813c3633b (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260430 07:57:02.071698 8870 tablet_service.cc:1511] Processing CreateTablet for tablet 37c58740f0c24968927cea27ad0d2948 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260430 07:57:02.072463 8869 tablet_service.cc:1511] Processing CreateTablet for tablet 6a14c736da5b465d8a0f8a7971a2840a (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260430 07:57:02.077560 8794 tablet_service.cc:1511] Processing CreateTablet for tablet 6a14c736da5b465d8a0f8a7971a2840a (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260430 07:57:02.096252 8793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8897623436482eb062dc3813c3633b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.095939 8867 tablet_service.cc:1511] Processing CreateTablet for tablet 1ad7efb0efd54c90a593c09fd7e2e320 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260430 07:57:02.096575 8866 tablet_service.cc:1511] Processing CreateTablet for tablet 936f73b154f54a41bfc33e3cfac79364 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260430 07:57:02.108734 8792 tablet_service.cc:1511] Processing CreateTablet for tablet 1ad7efb0efd54c90a593c09fd7e2e320 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260430 07:57:02.111780 8790 tablet_service.cc:1511] Processing CreateTablet for tablet 3a39a3eef27b4abaab420cdcbd358e2d (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260430 07:57:02.112056 8870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c58740f0c24968927cea27ad0d2948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.111565 8791 tablet_service.cc:1511] Processing CreateTablet for tablet 936f73b154f54a41bfc33e3cfac79364 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260430 07:57:02.099473 8868 tablet_service.cc:1511] Processing CreateTablet for tablet 8a8897623436482eb062dc3813c3633b (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260430 07:57:02.102669 8864 tablet_service.cc:1511] Processing CreateTablet for tablet 6d57fe9ae42243119707a7a47e4dccca (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260430 07:57:02.119451 8944 tablet_service.cc:1511] Processing CreateTablet for tablet 37c58740f0c24968927cea27ad0d2948 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260430 07:57:02.123270 8943 tablet_service.cc:1511] Processing CreateTablet for tablet 6a14c736da5b465d8a0f8a7971a2840a (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260430 07:57:02.125391 8943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a14c736da5b465d8a0f8a7971a2840a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.129932 8942 tablet_service.cc:1511] Processing CreateTablet for tablet 8a8897623436482eb062dc3813c3633b (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260430 07:57:02.131687 8792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad7efb0efd54c90a593c09fd7e2e320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.109521 8795 tablet_service.cc:1511] Processing CreateTablet for tablet 37c58740f0c24968927cea27ad0d2948 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260430 07:57:02.145486 8869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a14c736da5b465d8a0f8a7971a2840a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.131247 8865 tablet_service.cc:1511] Processing CreateTablet for tablet 3a39a3eef27b4abaab420cdcbd358e2d (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260430 07:57:02.135782 8789 tablet_service.cc:1511] Processing CreateTablet for tablet 6d57fe9ae42243119707a7a47e4dccca (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260430 07:57:02.172672 8794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a14c736da5b465d8a0f8a7971a2840a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.174472 8941 tablet_service.cc:1511] Processing CreateTablet for tablet 1ad7efb0efd54c90a593c09fd7e2e320 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260430 07:57:02.174675 8938 tablet_service.cc:1511] Processing CreateTablet for tablet 6d57fe9ae42243119707a7a47e4dccca (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260430 07:57:02.180724 8942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8897623436482eb062dc3813c3633b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.185549 8789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d57fe9ae42243119707a7a47e4dccca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.189477 8863 tablet_service.cc:1511] Processing CreateTablet for tablet d8a431f0e0bb4895aa5627ebc3ae1d79 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260430 07:57:02.188305 8868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8897623436482eb062dc3813c3633b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.184671 8937 tablet_service.cc:1511] Processing CreateTablet for tablet d8a431f0e0bb4895aa5627ebc3ae1d79 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260430 07:57:02.170629 8940 tablet_service.cc:1511] Processing CreateTablet for tablet 936f73b154f54a41bfc33e3cfac79364 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260430 07:57:02.179708 8939 tablet_service.cc:1511] Processing CreateTablet for tablet 3a39a3eef27b4abaab420cdcbd358e2d (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260430 07:57:02.197238 8944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c58740f0c24968927cea27ad0d2948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.204463 8791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936f73b154f54a41bfc33e3cfac79364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.179709 8788 tablet_service.cc:1511] Processing CreateTablet for tablet d8a431f0e0bb4895aa5627ebc3ae1d79 (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260430 07:57:02.207712 8787 tablet_service.cc:1511] Processing CreateTablet for tablet f55b0c696ea74901bb36afdd4d0e338c (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260430 07:57:02.210297 8865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a39a3eef27b4abaab420cdcbd358e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.220153 8938 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d57fe9ae42243119707a7a47e4dccca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.218557 8790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a39a3eef27b4abaab420cdcbd358e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.223965 8862 tablet_service.cc:1511] Processing CreateTablet for tablet f55b0c696ea74901bb36afdd4d0e338c (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260430 07:57:02.229864 8864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d57fe9ae42243119707a7a47e4dccca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.234710 8787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55b0c696ea74901bb36afdd4d0e338c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.237080 8788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8a431f0e0bb4895aa5627ebc3ae1d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.234953 8937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8a431f0e0bb4895aa5627ebc3ae1d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.241673 8862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55b0c696ea74901bb36afdd4d0e338c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.247030 8863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8a431f0e0bb4895aa5627ebc3ae1d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.250836 8939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a39a3eef27b4abaab420cdcbd358e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.250893 8866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936f73b154f54a41bfc33e3cfac79364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.256045 8795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c58740f0c24968927cea27ad0d2948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.254909 8936 tablet_service.cc:1511] Processing CreateTablet for tablet f55b0c696ea74901bb36afdd4d0e338c (DEFAULT_TABLE table=test-workload [id=8ce52c4e88704d0fbc6dfa66478319b4]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260430 07:57:02.264792 8867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad7efb0efd54c90a593c09fd7e2e320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.303278 8941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ad7efb0efd54c90a593c09fd7e2e320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.312460 8936 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55b0c696ea74901bb36afdd4d0e338c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.331863 8830 heartbeater.cc:499] Master 127.8.119.190:35737 was elected leader, sending a full tablet report...
I20260430 07:57:02.328953 8940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 936f73b154f54a41bfc33e3cfac79364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:02.538232 8997 tablet_bootstrap.cc:492] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad: Bootstrap starting.
I20260430 07:57:02.583885 8905 heartbeater.cc:499] Master 127.8.119.190:35737 was elected leader, sending a full tablet report...
I20260430 07:57:02.581538 8999 tablet_bootstrap.cc:492] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a: Bootstrap starting.
I20260430 07:57:02.584762 8998 tablet_bootstrap.cc:492] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52: Bootstrap starting.
I20260430 07:57:02.589144 8997 tablet_bootstrap.cc:654] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.614610 8999 tablet_bootstrap.cc:654] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.626663 8997 tablet_bootstrap.cc:492] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad: No bootstrap required, opened a new log
I20260430 07:57:02.627259 8997 ts_tablet_manager.cc:1403] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad: Time spent bootstrapping tablet: real 0.138s user 0.024s sys 0.015s
I20260430 07:57:02.627521 8998 tablet_bootstrap.cc:654] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.631526 8997 raft_consensus.cc:359] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.632759 8997 raft_consensus.cc:385] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:02.633391 8999 tablet_bootstrap.cc:492] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a: No bootstrap required, opened a new log
I20260430 07:57:02.633298 8997 raft_consensus.cc:740] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Initialized, Role: FOLLOWER
I20260430 07:57:02.634126 8999 ts_tablet_manager.cc:1403] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.012s
I20260430 07:57:02.634943 8997 consensus_queue.cc:260] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.638440 8999 raft_consensus.cc:359] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.639339 8999 raft_consensus.cc:385] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:02.639714 8999 raft_consensus.cc:740] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Initialized, Role: FOLLOWER
I20260430 07:57:02.640722 8999 consensus_queue.cc:260] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.646449 8998 tablet_bootstrap.cc:492] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52: No bootstrap required, opened a new log
I20260430 07:57:02.646996 8998 ts_tablet_manager.cc:1403] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.021s
I20260430 07:57:02.647078 8997 ts_tablet_manager.cc:1434] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad: Time spent starting tablet: real 0.019s user 0.018s sys 0.003s
I20260430 07:57:02.648203 8997 tablet_bootstrap.cc:492] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad: Bootstrap starting.
I20260430 07:57:02.650794 8998 raft_consensus.cc:359] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:02.651634 8998 raft_consensus.cc:385] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:02.652001 8998 raft_consensus.cc:740] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Initialized, Role: FOLLOWER
I20260430 07:57:02.654565 8999 ts_tablet_manager.cc:1434] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a: Time spent starting tablet: real 0.020s user 0.012s sys 0.002s
I20260430 07:57:02.655341 8979 heartbeater.cc:499] Master 127.8.119.190:35737 was elected leader, sending a full tablet report...
I20260430 07:57:02.655777 8999 tablet_bootstrap.cc:492] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a: Bootstrap starting.
I20260430 07:57:02.655728 8998 consensus_queue.cc:260] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:02.662246 9003 raft_consensus.cc:493] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:02.662932 9003 raft_consensus.cc:515] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.665778 8998 ts_tablet_manager.cc:1434] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52: Time spent starting tablet: real 0.018s user 0.010s sys 0.006s
I20260430 07:57:02.667469 8998 tablet_bootstrap.cc:492] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52: Bootstrap starting.
I20260430 07:57:02.674947 8997 tablet_bootstrap.cc:654] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.675029 8998 tablet_bootstrap.cc:654] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.675662 9003 leader_election.cc:290] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:02.692556 8999 tablet_bootstrap.cc:654] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.706445 8999 tablet_bootstrap.cc:492] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a: No bootstrap required, opened a new log
I20260430 07:57:02.706576 8998 tablet_bootstrap.cc:492] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52: No bootstrap required, opened a new log
I20260430 07:57:02.707131 8999 ts_tablet_manager.cc:1403] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a: Time spent bootstrapping tablet: real 0.052s user 0.020s sys 0.002s
I20260430 07:57:02.707273 8998 ts_tablet_manager.cc:1403] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.003s
I20260430 07:57:02.712106 8999 raft_consensus.cc:359] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.719182 8999 raft_consensus.cc:385] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:02.720176 8999 raft_consensus.cc:740] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Initialized, Role: FOLLOWER
I20260430 07:57:02.724555 8998 raft_consensus.cc:359] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.725685 8998 raft_consensus.cc:385] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:02.725993 8998 raft_consensus.cc:740] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Initialized, Role: FOLLOWER
I20260430 07:57:02.730105 8998 consensus_queue.cc:260] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.726632 8999 consensus_queue.cc:260] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.735212 8999 ts_tablet_manager.cc:1434] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20260430 07:57:02.735478 8997 tablet_bootstrap.cc:492] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad: No bootstrap required, opened a new log
I20260430 07:57:02.735991 8997 ts_tablet_manager.cc:1403] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad: Time spent bootstrapping tablet: real 0.088s user 0.023s sys 0.016s
I20260430 07:57:02.736397 8999 tablet_bootstrap.cc:492] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a: Bootstrap starting.
I20260430 07:57:02.740005 8998 ts_tablet_manager.cc:1434] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52: Time spent starting tablet: real 0.032s user 0.012s sys 0.004s
I20260430 07:57:02.741394 8998 tablet_bootstrap.cc:492] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52: Bootstrap starting.
I20260430 07:57:02.746682 8999 tablet_bootstrap.cc:654] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.750234 8998 tablet_bootstrap.cc:654] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.765879 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d57fe9ae42243119707a7a47e4dccca" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:02.768630 9005 raft_consensus.cc:493] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:02.740255 8997 raft_consensus.cc:359] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.769402 9005 raft_consensus.cc:515] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:02.770958 8997 raft_consensus.cc:385] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260430 07:57:02.771014 8831 tablet.cc:2404] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 07:57:02.771490 8997 raft_consensus.cc:740] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Initialized, Role: FOLLOWER
I20260430 07:57:02.772276 8997 consensus_queue.cc:260] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.777812 9005 leader_election.cc:290] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:02.782346 8997 ts_tablet_manager.cc:1434] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad: Time spent starting tablet: real 0.046s user 0.010s sys 0.001s
I20260430 07:57:02.784252 8997 tablet_bootstrap.cc:492] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad: Bootstrap starting.
I20260430 07:57:02.783738 8954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d57fe9ae42243119707a7a47e4dccca" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
I20260430 07:57:02.785641 8954 raft_consensus.cc:2468] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 0.
I20260430 07:57:02.788209 8771 leader_election.cc:304] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:02.793248 9003 raft_consensus.cc:2804] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:02.796772 9003 raft_consensus.cc:493] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:02.798542 9003 raft_consensus.cc:3060] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:57:02.797803 8770 leader_election.cc:343] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:02.820839 8998 tablet_bootstrap.cc:492] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52: No bootstrap required, opened a new log
I20260430 07:57:02.821413 8998 ts_tablet_manager.cc:1403] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52: Time spent bootstrapping tablet: real 0.080s user 0.018s sys 0.007s
W20260430 07:57:02.836774 8749 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:57:02.840901 8997 tablet_bootstrap.cc:654] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.845762 8998 raft_consensus.cc:359] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.850461 8998 raft_consensus.cc:385] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:02.851035 8998 raft_consensus.cc:740] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Initialized, Role: FOLLOWER
I20260430 07:57:02.844412 9003 raft_consensus.cc:515] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.856083 8999 tablet_bootstrap.cc:492] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a: No bootstrap required, opened a new log
I20260430 07:57:02.858800 8999 ts_tablet_manager.cc:1403] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a: Time spent bootstrapping tablet: real 0.123s user 0.012s sys 0.041s
I20260430 07:57:02.865770 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d57fe9ae42243119707a7a47e4dccca" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:02.863479 8998 consensus_queue.cc:260] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.870536 8998 ts_tablet_manager.cc:1434] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52: Time spent starting tablet: real 0.043s user 0.001s sys 0.009s
I20260430 07:57:02.864050 8954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d57fe9ae42243119707a7a47e4dccca" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:02.871245 8954 raft_consensus.cc:3060] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:57:02.870430 8770 leader_election.cc:343] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:02.871791 8998 tablet_bootstrap.cc:492] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52: Bootstrap starting.
I20260430 07:57:02.863363 9003 leader_election.cc:290] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 election: Requested vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:02.878121 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
W20260430 07:57:02.885102 8846 leader_election.cc:343] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:02.872229 8999 raft_consensus.cc:359] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.886574 8999 raft_consensus.cc:385] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:02.885856 9017 raft_consensus.cc:493] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:02.886915 8999 raft_consensus.cc:740] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Initialized, Role: FOLLOWER
I20260430 07:57:02.888815 9017 raft_consensus.cc:515] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.896215 8999 consensus_queue.cc:260] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.910208 8999 ts_tablet_manager.cc:1434] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a: Time spent starting tablet: real 0.050s user 0.011s sys 0.003s
I20260430 07:57:02.911319 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:02.912199 8880 raft_consensus.cc:2468] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 0.
I20260430 07:57:02.912955 8998 tablet_bootstrap.cc:654] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.908715 8997 tablet_bootstrap.cc:492] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad: No bootstrap required, opened a new log
I20260430 07:57:02.913847 8770 leader_election.cc:304] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
I20260430 07:57:02.915102 9003 raft_consensus.cc:2804] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:02.911331 8999 tablet_bootstrap.cc:492] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a: Bootstrap starting.
I20260430 07:57:02.915959 9003 raft_consensus.cc:493] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:02.916343 9003 raft_consensus.cc:3060] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:02.915342 8997 ts_tablet_manager.cc:1403] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad: Time spent bootstrapping tablet: real 0.131s user 0.018s sys 0.007s
I20260430 07:57:02.919968 8954 raft_consensus.cc:2468] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 1.
I20260430 07:57:02.922866 8771 leader_election.cc:304] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters: 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:02.924067 9017 leader_election.cc:290] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:02.926488 9017 raft_consensus.cc:2804] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:02.935313 9003 raft_consensus.cc:515] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.943140 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:02.926942 9017 raft_consensus.cc:697] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Becoming Leader. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
W20260430 07:57:02.948467 8845 leader_election.cc:343] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:02.949224 8845 leader_election.cc:304] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52; no voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:02.951215 9005 raft_consensus.cc:2749] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:02.958017 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:02.959100 8880 raft_consensus.cc:3060] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:02.959427 9003 leader_election.cc:290] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 election: Requested vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:02.963754 8997 raft_consensus.cc:359] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.966498 9017 consensus_queue.cc:237] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.968377 8999 tablet_bootstrap.cc:654] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:02.965873 8997 raft_consensus.cc:385] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:02.971278 8997 raft_consensus.cc:740] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Initialized, Role: FOLLOWER
I20260430 07:57:02.972796 8997 consensus_queue.cc:260] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
W20260430 07:57:02.970705 8771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:42455, user_credentials={real_user=slave}} blocked reactor thread for 47843.5us
I20260430 07:57:02.977059 8997 ts_tablet_manager.cc:1434] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad: Time spent starting tablet: real 0.055s user 0.001s sys 0.011s
I20260430 07:57:02.977263 8954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
I20260430 07:57:02.978665 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:02.975611 8880 raft_consensus.cc:2468] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 1.
W20260430 07:57:02.980465 8771 leader_election.cc:343] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:02.980707 8997 tablet_bootstrap.cc:492] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad: Bootstrap starting.
I20260430 07:57:02.980683 8770 leader_election.cc:304] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
W20260430 07:57:02.982371 8771 leader_election.cc:343] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:02.982517 9020 raft_consensus.cc:2804] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:02.983058 9020 raft_consensus.cc:697] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Becoming Leader. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:02.984463 9020 consensus_queue.cc:237] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:02.989113 8998 tablet_bootstrap.cc:492] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52: No bootstrap required, opened a new log
I20260430 07:57:02.989666 8998 ts_tablet_manager.cc:1403] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52: Time spent bootstrapping tablet: real 0.118s user 0.023s sys 0.023s
I20260430 07:57:02.995705 8999 tablet_bootstrap.cc:492] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a: No bootstrap required, opened a new log
I20260430 07:57:02.996318 8999 ts_tablet_manager.cc:1403] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a: Time spent bootstrapping tablet: real 0.085s user 0.014s sys 0.004s
I20260430 07:57:02.995627 8702 catalog_manager.cc:5671] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad reported cstate change: term changed from 0 to 1, leader changed from <none> to 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129). New cstate: current_term: 1 leader_uuid: "916b9fc1bae040e681e4b82126f3ecad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:02.999931 8998 raft_consensus.cc:359] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.004832 8998 raft_consensus.cc:385] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.006079 8998 raft_consensus.cc:740] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.007242 8998 consensus_queue.cc:260] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.002328 8997 tablet_bootstrap.cc:654] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.014061 8998 ts_tablet_manager.cc:1434] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52: Time spent starting tablet: real 0.024s user 0.006s sys 0.008s
I20260430 07:57:03.015177 8998 tablet_bootstrap.cc:492] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52: Bootstrap starting.
I20260430 07:57:02.999974 8999 raft_consensus.cc:359] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.017899 8999 raft_consensus.cc:385] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.018296 8999 raft_consensus.cc:740] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.019338 8999 consensus_queue.cc:260] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.022279 8999 ts_tablet_manager.cc:1434] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20260430 07:57:03.024008 8999 tablet_bootstrap.cc:492] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a: Bootstrap starting.
I20260430 07:57:03.040103 8997 tablet_bootstrap.cc:492] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad: No bootstrap required, opened a new log
I20260430 07:57:03.041704 8997 ts_tablet_manager.cc:1403] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.009s
I20260430 07:57:03.045719 8998 tablet_bootstrap.cc:654] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.051723 8999 tablet_bootstrap.cc:654] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.050465 8997 raft_consensus.cc:359] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.052472 8997 raft_consensus.cc:385] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.052799 8997 raft_consensus.cc:740] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.058173 9004 raft_consensus.cc:493] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.058410 8997 consensus_queue.cc:260] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.058894 9004 raft_consensus.cc:515] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.064285 9004 leader_election.cc:290] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.070801 8997 ts_tablet_manager.cc:1434] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad: Time spent starting tablet: real 0.029s user 0.013s sys 0.002s
I20260430 07:57:03.082615 8997 tablet_bootstrap.cc:492] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad: Bootstrap starting.
I20260430 07:57:03.086314 8998 tablet_bootstrap.cc:492] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52: No bootstrap required, opened a new log
I20260430 07:57:03.087100 8998 ts_tablet_manager.cc:1403] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.011s
I20260430 07:57:03.091612 8999 tablet_bootstrap.cc:492] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a: No bootstrap required, opened a new log
I20260430 07:57:03.092453 8999 ts_tablet_manager.cc:1403] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a: Time spent bootstrapping tablet: real 0.069s user 0.024s sys 0.010s
I20260430 07:57:03.093482 8998 raft_consensus.cc:359] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.094882 8998 raft_consensus.cc:385] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.095216 8998 raft_consensus.cc:740] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.096323 8998 consensus_queue.cc:260] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.097040 8997 tablet_bootstrap.cc:654] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.097653 8999 raft_consensus.cc:359] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.104560 8999 raft_consensus.cc:385] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.105017 8999 raft_consensus.cc:740] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.106148 8999 consensus_queue.cc:260] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.099342 8998 ts_tablet_manager.cc:1434] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260430 07:57:03.111729 8998 tablet_bootstrap.cc:492] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52: Bootstrap starting.
I20260430 07:57:03.112564 8999 ts_tablet_manager.cc:1434] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260430 07:57:03.114298 8999 tablet_bootstrap.cc:492] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a: Bootstrap starting.
I20260430 07:57:03.119318 9004 raft_consensus.cc:493] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.122687 8998 tablet_bootstrap.cc:654] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.119850 9004 raft_consensus.cc:515] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.127386 9004 leader_election.cc:290] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.148336 8879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:03.146811 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:03.150118 8880 raft_consensus.cc:2468] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 0.
I20260430 07:57:03.145512 8702 catalog_manager.cc:5671] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad reported cstate change: term changed from 0 to 1, leader changed from <none> to 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129). New cstate: current_term: 1 leader_uuid: "916b9fc1bae040e681e4b82126f3ecad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
W20260430 07:57:03.154057 8919 leader_election.cc:343] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.155360 8919 leader_election.cc:304] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:03.155897 8997 tablet_bootstrap.cc:492] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad: No bootstrap required, opened a new log
I20260430 07:57:03.156832 9004 raft_consensus.cc:2804] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:03.157894 9004 raft_consensus.cc:493] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:03.158212 9004 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.156862 8997 ts_tablet_manager.cc:1403] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.008s
I20260430 07:57:03.166344 8999 tablet_bootstrap.cc:654] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.170096 8998 tablet_bootstrap.cc:492] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52: No bootstrap required, opened a new log
I20260430 07:57:03.171689 8998 ts_tablet_manager.cc:1403] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52: Time spent bootstrapping tablet: real 0.060s user 0.025s sys 0.005s
I20260430 07:57:03.175618 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:03.178648 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:03.177904 8805 raft_consensus.cc:2468] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 0.
I20260430 07:57:03.181161 8804 raft_consensus.cc:2468] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 0.
I20260430 07:57:03.183075 8919 leader_election.cc:304] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters: 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:03.186921 9004 raft_consensus.cc:515] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.194923 8998 raft_consensus.cc:359] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.186931 8997 raft_consensus.cc:359] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.197220 8997 raft_consensus.cc:385] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.197580 8997 raft_consensus.cc:740] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.197820 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:03.198589 8880 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.197822 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:03.199249 8804 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.195382 9004 leader_election.cc:290] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.198414 8997 consensus_queue.cc:260] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.195855 9036 raft_consensus.cc:2804] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:03.196328 8998 raft_consensus.cc:385] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.207682 9036 raft_consensus.cc:493] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:03.208151 9036 raft_consensus.cc:3060] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.205024 8997 ts_tablet_manager.cc:1434] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad: Time spent starting tablet: real 0.045s user 0.008s sys 0.002s
I20260430 07:57:03.210038 8997 tablet_bootstrap.cc:492] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad: Bootstrap starting.
I20260430 07:57:03.207697 8998 raft_consensus.cc:740] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.218192 8998 consensus_queue.cc:260] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.208303 8880 raft_consensus.cc:2468] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 1.
I20260430 07:57:03.219249 8999 tablet_bootstrap.cc:492] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a: No bootstrap required, opened a new log
I20260430 07:57:03.219883 8999 ts_tablet_manager.cc:1403] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a: Time spent bootstrapping tablet: real 0.106s user 0.014s sys 0.009s
I20260430 07:57:03.220058 8919 leader_election.cc:304] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:03.220974 9004 raft_consensus.cc:2804] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:03.223589 8999 raft_consensus.cc:359] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.224798 9036 raft_consensus.cc:515] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.224762 8804 raft_consensus.cc:2468] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 1.
I20260430 07:57:03.225378 8999 raft_consensus.cc:385] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.226636 8999 raft_consensus.cc:740] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.227206 9036 leader_election.cc:290] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.227872 8999 consensus_queue.cc:260] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.229724 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:03.230510 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:03.230386 8804 raft_consensus.cc:3060] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.231523 8999 ts_tablet_manager.cc:1434] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
W20260430 07:57:03.231917 8919 leader_election.cc:343] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.228647 9036 raft_consensus.cc:493] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.233304 9036 raft_consensus.cc:515] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.235437 9036 leader_election.cc:290] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.237051 8999 tablet_bootstrap.cc:492] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a: Bootstrap starting.
I20260430 07:57:03.237243 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:03.240895 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:03.241775 8880 raft_consensus.cc:2393] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec08915611e344558a04e8a9c11b6b0a in current term 1: Already voted for candidate 916b9fc1bae040e681e4b82126f3ecad in this term.
I20260430 07:57:03.242733 8919 leader_election.cc:304] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec08915611e344558a04e8a9c11b6b0a; no voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad
I20260430 07:57:03.243803 8998 ts_tablet_manager.cc:1434] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52: Time spent starting tablet: real 0.072s user 0.005s sys 0.004s
I20260430 07:57:03.244376 9036 raft_consensus.cc:3060] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.245810 8999 tablet_bootstrap.cc:654] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.251662 8804 raft_consensus.cc:2468] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 1.
I20260430 07:57:03.254683 9039 raft_consensus.cc:493] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.255474 9039 raft_consensus.cc:515] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.256032 8919 leader_election.cc:304] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters: 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:03.257498 9038 raft_consensus.cc:2804] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:03.257673 9039 leader_election.cc:290] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.257925 8998 tablet_bootstrap.cc:492] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52: Bootstrap starting.
I20260430 07:57:03.258224 9038 raft_consensus.cc:697] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Becoming Leader. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:03.259372 9038 consensus_queue.cc:237] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.261647 9004 raft_consensus.cc:697] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Becoming Leader. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:03.265133 9004 consensus_queue.cc:237] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.267848 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a8897623436482eb062dc3813c3633b" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:03.269106 8880 raft_consensus.cc:2468] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 0.
I20260430 07:57:03.270742 8997 tablet_bootstrap.cc:654] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.271327 8919 leader_election.cc:304] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:03.267819 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a8897623436482eb062dc3813c3633b" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:03.274926 9036 raft_consensus.cc:2749] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:03.273758 9039 raft_consensus.cc:2804] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:03.276690 9039 raft_consensus.cc:493] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:03.277263 9039 raft_consensus.cc:3060] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:57:03.277959 8919 leader_election.cc:343] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.280503 8999 tablet_bootstrap.cc:492] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a: No bootstrap required, opened a new log
I20260430 07:57:03.282878 8999 ts_tablet_manager.cc:1403] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a: Time spent bootstrapping tablet: real 0.050s user 0.023s sys 0.008s
I20260430 07:57:03.284353 8998 tablet_bootstrap.cc:654] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.285845 8700 catalog_manager.cc:5671] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to ec08915611e344558a04e8a9c11b6b0a (127.8.119.131). New cstate: current_term: 1 leader_uuid: "ec08915611e344558a04e8a9c11b6b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:03.292584 9005 raft_consensus.cc:493] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.293133 9005 raft_consensus.cc:515] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.297091 9005 leader_election.cc:290] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:03.297712 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d57fe9ae42243119707a7a47e4dccca" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:03.297411 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d57fe9ae42243119707a7a47e4dccca" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
I20260430 07:57:03.304891 8953 raft_consensus.cc:2393] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35eb09ea173d4264b201b528036d5b52 in current term 1: Already voted for candidate 916b9fc1bae040e681e4b82126f3ecad in this term.
I20260430 07:57:03.308183 8999 raft_consensus.cc:359] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:03.308187 8846 leader_election.cc:304] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52; no voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:03.311007 8999 raft_consensus.cc:385] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.311395 8999 raft_consensus.cc:740] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.311455 9005 raft_consensus.cc:3060] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.312214 9039 raft_consensus.cc:515] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.319960 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a8897623436482eb062dc3813c3633b" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:03.321125 9039 leader_election.cc:290] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.321511 8998 tablet_bootstrap.cc:492] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52: No bootstrap required, opened a new log
I20260430 07:57:03.321995 8998 ts_tablet_manager.cc:1403] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52: Time spent bootstrapping tablet: real 0.064s user 0.014s sys 0.005s
I20260430 07:57:03.322256 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a8897623436482eb062dc3813c3633b" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52"
W20260430 07:57:03.322479 8919 leader_election.cc:343] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.323483 8880 raft_consensus.cc:3060] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.326236 9005 raft_consensus.cc:2749] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:03.325893 8998 raft_consensus.cc:359] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.328879 8998 raft_consensus.cc:385] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.329224 8998 raft_consensus.cc:740] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.314767 8999 consensus_queue.cc:260] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:03.327942 8997 tablet_bootstrap.cc:492] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad: No bootstrap required, opened a new log
I20260430 07:57:03.330721 8997 ts_tablet_manager.cc:1403] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad: Time spent bootstrapping tablet: real 0.121s user 0.012s sys 0.016s
I20260430 07:57:03.332079 8999 ts_tablet_manager.cc:1434] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a: Time spent starting tablet: real 0.048s user 0.004s sys 0.004s
I20260430 07:57:03.332201 8880 raft_consensus.cc:2468] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 1.
I20260430 07:57:03.333808 8998 consensus_queue.cc:260] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.332468 8700 catalog_manager.cc:5671] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to ec08915611e344558a04e8a9c11b6b0a (127.8.119.131). New cstate: current_term: 1 leader_uuid: "ec08915611e344558a04e8a9c11b6b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:03.339174 9005 raft_consensus.cc:493] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.334394 8999 tablet_bootstrap.cc:492] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a: Bootstrap starting.
I20260430 07:57:03.339711 9005 raft_consensus.cc:515] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.342998 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
W20260430 07:57:03.344590 8845 leader_election.cc:343] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.345783 9005 leader_election.cc:290] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:03.347530 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
W20260430 07:57:03.349344 8846 leader_election.cc:343] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.334395 8997 raft_consensus.cc:359] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.349965 8846 leader_election.cc:304] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52; no voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:03.350515 8997 raft_consensus.cc:385] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.351246 8997 raft_consensus.cc:740] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.352099 8997 consensus_queue.cc:260] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.354001 9005 raft_consensus.cc:2749] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:03.355504 8997 ts_tablet_manager.cc:1434] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad: Time spent starting tablet: real 0.024s user 0.004s sys 0.003s
I20260430 07:57:03.355945 8919 leader_election.cc:304] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters: 916b9fc1bae040e681e4b82126f3ecad
I20260430 07:57:03.356714 8997 tablet_bootstrap.cc:492] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad: Bootstrap starting.
I20260430 07:57:03.357470 9039 raft_consensus.cc:2804] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:03.358161 9039 raft_consensus.cc:697] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Becoming Leader. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:03.359182 9039 consensus_queue.cc:237] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.364084 8999 tablet_bootstrap.cc:654] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.366660 8997 tablet_bootstrap.cc:654] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.374125 9038 raft_consensus.cc:493] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.374925 9038 raft_consensus.cc:515] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.379733 8997 tablet_bootstrap.cc:492] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad: No bootstrap required, opened a new log
I20260430 07:57:03.383706 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:03.384270 8999 tablet_bootstrap.cc:492] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a: No bootstrap required, opened a new log
I20260430 07:57:03.384845 8999 ts_tablet_manager.cc:1403] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.016s
I20260430 07:57:03.384622 8804 raft_consensus.cc:2468] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 0.
I20260430 07:57:03.387044 8997 ts_tablet_manager.cc:1403] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.000s
I20260430 07:57:03.387123 9020 consensus_queue.cc:1048] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:03.388135 9038 leader_election.cc:290] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.388546 8999 raft_consensus.cc:359] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.389039 8919 leader_election.cc:304] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:03.390278 8997 raft_consensus.cc:359] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:03.391090 8997 raft_consensus.cc:385] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.391443 8997 raft_consensus.cc:740] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.392136 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:03.392234 8997 consensus_queue.cc:260] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:03.395328 8997 ts_tablet_manager.cc:1434] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20260430 07:57:03.396445 8997 tablet_bootstrap.cc:492] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad: Bootstrap starting.
I20260430 07:57:03.396332 9020 raft_consensus.cc:493] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.397019 9020 raft_consensus.cc:515] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.396086 8998 ts_tablet_manager.cc:1434] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52: Time spent starting tablet: real 0.074s user 0.011s sys 0.011s
I20260430 07:57:03.399649 9017 raft_consensus.cc:493] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.400839 9020 leader_election.cc:290] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:03.389231 8999 raft_consensus.cc:385] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260430 07:57:03.404175 8919 leader_election.cc:343] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.407958 9038 raft_consensus.cc:2804] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:03.409452 9038 raft_consensus.cc:493] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:03.402119 8879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f55b0c696ea74901bb36afdd4d0e338c" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:03.400835 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f55b0c696ea74901bb36afdd4d0e338c" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
I20260430 07:57:03.405450 8999 raft_consensus.cc:740] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.400866 9017 raft_consensus.cc:515] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:03.409180 9020 consensus_queue.cc:1048] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:03.411942 8999 consensus_queue.cc:260] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
W20260430 07:57:03.412732 8770 leader_election.cc:343] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.412868 9017 leader_election.cc:290] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.414482 8999 ts_tablet_manager.cc:1434] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a: Time spent starting tablet: real 0.029s user 0.009s sys 0.000s
I20260430 07:57:03.414999 9038 raft_consensus.cc:3060] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.415202 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
I20260430 07:57:03.415762 8999 tablet_bootstrap.cc:492] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a: Bootstrap starting.
I20260430 07:57:03.415820 8953 raft_consensus.cc:2468] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 0.
W20260430 07:57:03.414418 8771 leader_election.cc:343] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.416749 8771 leader_election.cc:304] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad; no voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:03.417866 9017 raft_consensus.cc:2749] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:03.399389 8998 tablet_bootstrap.cc:492] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52: Bootstrap starting.
I20260430 07:57:03.414361 8879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:03.425465 8879 raft_consensus.cc:2468] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 0.
I20260430 07:57:03.410176 8997 tablet_bootstrap.cc:654] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.419821 8771 leader_election.cc:304] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:03.430730 9020 raft_consensus.cc:2804] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:03.431139 9020 raft_consensus.cc:493] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:03.431468 9020 raft_consensus.cc:3060] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.448642 9038 raft_consensus.cc:515] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.453118 8999 tablet_bootstrap.cc:654] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.452903 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:03.453382 9038 leader_election.cc:290] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.453608 8879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:03.453624 8804 raft_consensus.cc:3060] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:57:03.455404 8919 leader_election.cc:343] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.459741 8998 tablet_bootstrap.cc:654] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.462376 9020 raft_consensus.cc:515] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:03.460016 9038 raft_consensus.cc:493] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.476830 9020 leader_election.cc:290] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 election: Requested vote from peers ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.478914 8879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:03.479616 8879 raft_consensus.cc:3060] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.469796 9038 raft_consensus.cc:515] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.489965 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:03.491839 8997 tablet_bootstrap.cc:492] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad: No bootstrap required, opened a new log
W20260430 07:57:03.492159 8919 leader_election.cc:343] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.492331 8997 ts_tablet_manager.cc:1403] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad: Time spent bootstrapping tablet: real 0.096s user 0.006s sys 0.013s
I20260430 07:57:03.496023 9017 consensus_queue.cc:1048] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260430 07:57:03.499722 8997 raft_consensus.cc:359] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.500876 8997 raft_consensus.cc:385] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.501214 8997 raft_consensus.cc:740] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.501955 8997 consensus_queue.cc:260] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.507229 8997 ts_tablet_manager.cc:1434] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad: Time spent starting tablet: real 0.014s user 0.007s sys 0.002s
I20260430 07:57:03.502719 8804 raft_consensus.cc:2468] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 1.
I20260430 07:57:03.504177 9038 leader_election.cc:290] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:03.510623 9017 raft_consensus.cc:493] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.518126 8919 leader_election.cc:304] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters: 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:03.518229 9017 raft_consensus.cc:515] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.519534 9038 raft_consensus.cc:2804] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:03.521602 9038 raft_consensus.cc:697] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Becoming Leader. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:03.510037 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:03.530817 8952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a8897623436482eb062dc3813c3633b" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
I20260430 07:57:03.530854 8953 raft_consensus.cc:3060] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.542673 8878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:03.545316 8878 raft_consensus.cc:2468] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 0.
I20260430 07:57:03.547442 9038 consensus_queue.cc:237] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.521168 9017 leader_election.cc:290] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:03.556103 8919 leader_election.cc:304] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters: 916b9fc1bae040e681e4b82126f3ecad
I20260430 07:57:03.568969 8998 tablet_bootstrap.cc:492] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52: No bootstrap required, opened a new log
I20260430 07:57:03.569507 8998 ts_tablet_manager.cc:1403] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52: Time spent bootstrapping tablet: real 0.170s user 0.014s sys 0.007s
I20260430 07:57:03.573185 8998 raft_consensus.cc:359] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.578006 8998 raft_consensus.cc:385] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.578541 8998 raft_consensus.cc:740] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.579329 8997 tablet_bootstrap.cc:492] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad: Bootstrap starting.
I20260430 07:57:03.579416 8998 consensus_queue.cc:260] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.570358 9004 raft_consensus.cc:2804] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:03.593245 9045 raft_consensus.cc:493] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.594417 9004 raft_consensus.cc:493] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:03.594964 9004 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.593837 9045 raft_consensus.cc:515] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.614919 9045 leader_election.cc:290] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:03.617434 8998 ts_tablet_manager.cc:1434] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52: Time spent starting tablet: real 0.048s user 0.008s sys 0.004s
I20260430 07:57:03.618813 8998 tablet_bootstrap.cc:492] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52: Bootstrap starting.
I20260430 07:57:03.627209 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:03.630678 8804 raft_consensus.cc:2393] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35eb09ea173d4264b201b528036d5b52 in current term 1: Already voted for candidate ec08915611e344558a04e8a9c11b6b0a in this term.
I20260430 07:57:03.619622 8879 raft_consensus.cc:2468] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 1.
I20260430 07:57:03.638265 8951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
W20260430 07:57:03.658001 9048 log.cc:927] Time spent T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260430 07:57:03.654719 8846 leader_election.cc:304] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52; no voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:03.686061 8879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a8897623436482eb062dc3813c3633b" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:03.692181 8879 raft_consensus.cc:2393] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 916b9fc1bae040e681e4b82126f3ecad in current term 1: Already voted for candidate ec08915611e344558a04e8a9c11b6b0a in this term.
I20260430 07:57:03.692531 9005 raft_consensus.cc:3060] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.687832 9045 raft_consensus.cc:493] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:03.705190 9045 raft_consensus.cc:515] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.737059 9004 raft_consensus.cc:515] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.772701 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:03.791447 9004 leader_election.cc:290] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
W20260430 07:57:03.794157 8919 leader_election.cc:343] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:03.800915 9045 leader_election.cc:290] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:03.801823 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:03.802637 8804 raft_consensus.cc:2393] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35eb09ea173d4264b201b528036d5b52 in current term 1: Already voted for candidate ec08915611e344558a04e8a9c11b6b0a in this term.
I20260430 07:57:03.802628 8879 raft_consensus.cc:3060] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.808025 8953 raft_consensus.cc:2468] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 1.
I20260430 07:57:03.808267 8878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:03.809018 8878 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.809545 9005 raft_consensus.cc:2749] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:03.726483 8770 leader_election.cc:304] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
W20260430 07:57:03.810421 8770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:39379, user_credentials={real_user=slave}} blocked reactor thread for 88887.2us
I20260430 07:57:03.811151 8770 leader_election.cc:304] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad; no voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:03.812433 9017 raft_consensus.cc:2804] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:03.814007 9055 raft_consensus.cc:3060] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:03.813997 9017 raft_consensus.cc:697] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Becoming Leader. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:03.815770 9017 consensus_queue.cc:237] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:03.821130 9020 consensus_queue.cc:1048] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260430 07:57:03.838055 8999 tablet_bootstrap.cc:492] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a: No bootstrap required, opened a new log
I20260430 07:57:03.838896 8999 ts_tablet_manager.cc:1403] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a: Time spent bootstrapping tablet: real 0.423s user 0.034s sys 0.020s
I20260430 07:57:03.847429 8878 raft_consensus.cc:2468] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 1.
I20260430 07:57:03.853531 8999 raft_consensus.cc:359] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.854919 8999 raft_consensus.cc:385] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.855386 8999 raft_consensus.cc:740] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Initialized, Role: FOLLOWER
W20260430 07:57:03.857913 8749 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:57:03.860908 8919 leader_election.cc:304] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters: 916b9fc1bae040e681e4b82126f3ecad
I20260430 07:57:03.864316 8998 tablet_bootstrap.cc:654] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.864208 9039 raft_consensus.cc:2804] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:03.870857 8999 consensus_queue.cc:260] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.873765 8997 tablet_bootstrap.cc:654] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:03.852356 8952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
I20260430 07:57:03.887632 8999 ts_tablet_manager.cc:1434] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a: Time spent starting tablet: real 0.048s user 0.010s sys 0.001s
I20260430 07:57:03.890409 9004 consensus_queue.cc:1048] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260430 07:57:03.889763 9039 raft_consensus.cc:697] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Becoming Leader. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:03.898779 8846 leader_election.cc:304] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52; no voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:03.894816 8699 catalog_manager.cc:5671] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to ec08915611e344558a04e8a9c11b6b0a (127.8.119.131). New cstate: current_term: 1 leader_uuid: "ec08915611e344558a04e8a9c11b6b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:03.908483 9061 raft_consensus.cc:2749] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:03.908695 9065 consensus_queue.cc:1048] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:03.921944 9055 raft_consensus.cc:2749] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:03.924310 8998 tablet_bootstrap.cc:492] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52: No bootstrap required, opened a new log
I20260430 07:57:03.925036 8998 ts_tablet_manager.cc:1403] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52: Time spent bootstrapping tablet: real 0.306s user 0.020s sys 0.000s
I20260430 07:57:03.926944 9039 consensus_queue.cc:237] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.928663 9038 consensus_queue.cc:1048] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:03.934924 8998 raft_consensus.cc:359] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.935796 8998 raft_consensus.cc:385] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.935400 9066 consensus_queue.cc:1048] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:03.936630 8998 raft_consensus.cc:740] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.932875 9038 consensus_queue.cc:1048] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:03.937441 8998 consensus_queue.cc:260] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.942504 8998 ts_tablet_manager.cc:1434] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260430 07:57:03.962779 8997 tablet_bootstrap.cc:492] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad: No bootstrap required, opened a new log
I20260430 07:57:03.963369 8997 ts_tablet_manager.cc:1403] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad: Time spent bootstrapping tablet: real 0.384s user 0.023s sys 0.011s
I20260430 07:57:03.972292 8997 raft_consensus.cc:359] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.974479 8997 raft_consensus.cc:385] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:03.974803 8997 raft_consensus.cc:740] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Initialized, Role: FOLLOWER
I20260430 07:57:03.976323 8997 consensus_queue.cc:260] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:03.983068 8997 ts_tablet_manager.cc:1434] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad: Time spent starting tablet: real 0.019s user 0.005s sys 0.002s
I20260430 07:57:03.983973 9066 consensus_queue.cc:1048] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:04.007504 9061 raft_consensus.cc:493] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:04.008062 9061 raft_consensus.cc:515] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.010915 9061 leader_election.cc:290] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:04.010849 8700 catalog_manager.cc:5671] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad reported cstate change: term changed from 0 to 1, leader changed from <none> to 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129). New cstate: current_term: 1 leader_uuid: "916b9fc1bae040e681e4b82126f3ecad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:04.013609 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f55b0c696ea74901bb36afdd4d0e338c" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad" is_pre_election: true
I20260430 07:57:04.014324 8804 raft_consensus.cc:2468] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 0.
I20260430 07:57:04.013602 8952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f55b0c696ea74901bb36afdd4d0e338c" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
I20260430 07:57:04.016085 8952 raft_consensus.cc:2468] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 0.
I20260430 07:57:04.018554 8845 leader_election.cc:304] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
I20260430 07:57:04.019344 9061 raft_consensus.cc:2804] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:04.019927 9061 raft_consensus.cc:493] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:04.020378 9061 raft_consensus.cc:3060] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:04.035145 9061 raft_consensus.cc:515] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.040243 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f55b0c696ea74901bb36afdd4d0e338c" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:04.041392 8805 raft_consensus.cc:3060] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:04.050674 9061 leader_election.cc:290] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:04.051478 8805 raft_consensus.cc:2468] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 1.
I20260430 07:57:04.051476 8952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f55b0c696ea74901bb36afdd4d0e338c" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:04.052209 8952 raft_consensus.cc:3060] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:04.054191 8845 leader_election.cc:304] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
I20260430 07:57:04.055738 9061 raft_consensus.cc:2804] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:04.054616 8702 catalog_manager.cc:5671] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to ec08915611e344558a04e8a9c11b6b0a (127.8.119.131). New cstate: current_term: 1 leader_uuid: "ec08915611e344558a04e8a9c11b6b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:04.056609 9061 raft_consensus.cc:697] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 1 LEADER]: Becoming Leader. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:04.057161 8702 catalog_manager.cc:5671] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to ec08915611e344558a04e8a9c11b6b0a (127.8.119.131). New cstate: current_term: 1 leader_uuid: "ec08915611e344558a04e8a9c11b6b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:04.071185 9061 consensus_queue.cc:237] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.076449 8952 raft_consensus.cc:2468] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 1.
I20260430 07:57:04.097209 8700 catalog_manager.cc:5671] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35eb09ea173d4264b201b528036d5b52 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "35eb09ea173d4264b201b528036d5b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:04.183763 9038 consensus_queue.cc:1048] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:04.199074 9066 consensus_queue.cc:1048] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:04.220767 9020 consensus_queue.cc:1048] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:04.224377 9020 raft_consensus.cc:493] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:04.225060 9020 raft_consensus.cc:515] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.230415 9020 leader_election.cc:290] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:04.230680 8952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec08915611e344558a04e8a9c11b6b0a" is_pre_election: true
I20260430 07:57:04.230680 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35eb09ea173d4264b201b528036d5b52" is_pre_election: true
I20260430 07:57:04.231815 8880 raft_consensus.cc:2393] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 916b9fc1bae040e681e4b82126f3ecad in current term 1: Already voted for candidate ec08915611e344558a04e8a9c11b6b0a in this term.
I20260430 07:57:04.233739 8770 leader_election.cc:304] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad; no voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:04.239028 9020 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:04.253928 9055 consensus_queue.cc:1048] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260430 07:57:04.273486 9020 raft_consensus.cc:2749] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:04.313241 8952 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
mode: GRACEFUL
new_leader_uuid: "916b9fc1bae040e681e4b82126f3ecad"
from {username='slave'} at 127.0.0.1:59388
I20260430 07:57:04.314023 8952 raft_consensus.cc:606] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Received request to transfer leadership to TS 916b9fc1bae040e681e4b82126f3ecad
I20260430 07:57:04.320955 8952 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8a8897623436482eb062dc3813c3633b"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
mode: GRACEFUL
new_leader_uuid: "916b9fc1bae040e681e4b82126f3ecad"
from {username='slave'} at 127.0.0.1:59388
I20260430 07:57:04.321681 8952 raft_consensus.cc:606] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Received request to transfer leadership to TS 916b9fc1bae040e681e4b82126f3ecad
I20260430 07:57:04.332271 8952 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
mode: GRACEFUL
new_leader_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:59388
I20260430 07:57:04.332907 8952 raft_consensus.cc:606] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Received request to transfer leadership to TS 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:04.341657 8952 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
mode: GRACEFUL
new_leader_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:59388
I20260430 07:57:04.343432 8952 raft_consensus.cc:606] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Received request to transfer leadership to TS 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:04.368824 8805 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6d57fe9ae42243119707a7a47e4dccca"
dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
mode: GRACEFUL
new_leader_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:59212
I20260430 07:57:04.369442 8805 raft_consensus.cc:606] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Received request to transfer leadership to TS 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:04.375563 8952 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
mode: GRACEFUL
new_leader_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:59388
I20260430 07:57:04.376506 8952 raft_consensus.cc:606] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Received request to transfer leadership to TS 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:04.382208 9038 consensus_queue.cc:1048] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:04.398101 8877 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f55b0c696ea74901bb36afdd4d0e338c"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
mode: GRACEFUL
new_leader_uuid: "ec08915611e344558a04e8a9c11b6b0a"
from {username='slave'} at 127.0.0.1:53800
I20260430 07:57:04.398728 8877 raft_consensus.cc:606] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 1 LEADER]: Received request to transfer leadership to TS ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:04.406529 9038 consensus_queue.cc:1048] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:04.419960 9066 raft_consensus.cc:993] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a: : Instructing follower 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:04.420521 9066 raft_consensus.cc:1081] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Signalling peer 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:04.423621 8877 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:53790
I20260430 07:57:04.424394 8877 raft_consensus.cc:493] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:04.424880 8877 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.426467 9065 raft_consensus.cc:993] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a: : Instructing follower 916b9fc1bae040e681e4b82126f3ecad to start an election
I20260430 07:57:04.426980 9038 raft_consensus.cc:1081] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Signalling peer 916b9fc1bae040e681e4b82126f3ecad to start an election
I20260430 07:57:04.429517 8805 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8a8897623436482eb062dc3813c3633b"
dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
from {username='slave'} at 127.0.0.1:59206
I20260430 07:57:04.430157 8805 raft_consensus.cc:493] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:04.430555 8805 raft_consensus.cc:3060] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.439459 8805 raft_consensus.cc:515] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.439483 8877 raft_consensus.cc:515] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.443748 8805 leader_election.cc:290] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 2 election: Requested vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:04.444854 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a8897623436482eb062dc3813c3633b" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:04.445654 8880 raft_consensus.cc:3060] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.447080 8952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a8897623436482eb062dc3813c3633b" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:04.447759 8952 raft_consensus.cc:3055] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:04.448098 8952 raft_consensus.cc:740] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:04.448925 8952 consensus_queue.cc:260] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.450455 8952 raft_consensus.cc:3060] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.452584 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:04.456001 8804 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.530110 9065 raft_consensus.cc:993] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a: : Instructing follower 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:04.530788 9039 raft_consensus.cc:1081] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Signalling peer 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:04.533430 8879 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:53790
I20260430 07:57:04.534243 8879 raft_consensus.cc:493] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:04.535267 8879 raft_consensus.cc:3060] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.547575 9061 consensus_queue.cc:1048] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:04.601140 8880 raft_consensus.cc:2468] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 2.
I20260430 07:57:04.602941 8877 leader_election.cc:290] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 2 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:04.602735 8770 leader_election.cc:304] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
I20260430 07:57:04.604645 9055 raft_consensus.cc:2804] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:04.605193 9055 raft_consensus.cc:697] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 2 LEADER]: Becoming Leader. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:04.606051 9055 consensus_queue.cc:237] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.607437 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:04.608516 8953 raft_consensus.cc:3055] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:04.608943 8953 raft_consensus.cc:740] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:04.609788 8953 consensus_queue.cc:260] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.611086 8953 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.613606 8952 raft_consensus.cc:2468] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 2.
I20260430 07:57:04.622992 8804 raft_consensus.cc:2468] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 2.
I20260430 07:57:04.624267 8845 leader_election.cc:304] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
I20260430 07:57:04.625710 9061 raft_consensus.cc:2804] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:04.634559 9065 raft_consensus.cc:993] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a: : Instructing follower 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:04.635219 8953 raft_consensus.cc:2468] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 2.
I20260430 07:57:04.635053 9039 raft_consensus.cc:1081] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Signalling peer 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:04.638669 8877 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:53790
I20260430 07:57:04.640554 8877 raft_consensus.cc:493] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:04.641023 8877 raft_consensus.cc:3060] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.662971 8877 raft_consensus.cc:515] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.665804 8877 leader_election.cc:290] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 2 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:04.670325 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:04.671065 8953 raft_consensus.cc:3055] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:04.671428 8953 raft_consensus.cc:740] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:04.672274 8953 consensus_queue.cc:260] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.673750 8953 raft_consensus.cc:3060] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.690858 8953 raft_consensus.cc:2468] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 2.
I20260430 07:57:04.700064 9061 raft_consensus.cc:697] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Becoming Leader. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:04.701797 9061 consensus_queue.cc:237] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.863488 8846 leader_election.cc:304] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:04.865775 9092 raft_consensus.cc:2804] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:04.868096 9092 raft_consensus.cc:697] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Becoming Leader. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:04.869187 9092 consensus_queue.cc:237] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.867209 9091 raft_consensus.cc:993] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52: : Instructing follower ec08915611e344558a04e8a9c11b6b0a to start an election
I20260430 07:57:04.873859 9091 raft_consensus.cc:1081] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 1 LEADER]: Signalling peer ec08915611e344558a04e8a9c11b6b0a to start an election
I20260430 07:57:04.881551 9092 consensus_queue.cc:1048] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:04.902906 9017 raft_consensus.cc:993] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad: : Instructing follower 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:04.903743 9017 raft_consensus.cc:1081] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Signalling peer 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:04.909309 8877 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6d57fe9ae42243119707a7a47e4dccca"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:53774
I20260430 07:57:04.909996 8877 raft_consensus.cc:493] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:04.910415 8877 raft_consensus.cc:3060] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.937716 8954 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f55b0c696ea74901bb36afdd4d0e338c"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
from {username='slave'} at 127.0.0.1:59376
I20260430 07:57:04.938980 8954 raft_consensus.cc:493] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:04.939430 8954 raft_consensus.cc:3060] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.939989 9038 raft_consensus.cc:993] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a: : Instructing follower 916b9fc1bae040e681e4b82126f3ecad to start an election
I20260430 07:57:04.940428 9038 raft_consensus.cc:1081] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Signalling peer 916b9fc1bae040e681e4b82126f3ecad to start an election
I20260430 07:57:04.946451 8805 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a"
dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
from {username='slave'} at 127.0.0.1:59206
I20260430 07:57:04.947079 8805 raft_consensus.cc:493] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:04.947460 8805 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.948340 8804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:04.948987 8804 raft_consensus.cc:3060] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.961544 8805 raft_consensus.cc:515] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.966137 8805 leader_election.cc:290] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 2 election: Requested vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:04.968078 8880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:04.968735 8880 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.969696 8952 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:04.970412 8952 raft_consensus.cc:3055] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:04.970795 8952 raft_consensus.cc:740] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:04.972157 8952 consensus_queue.cc:260] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.973409 8877 raft_consensus.cc:515] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.981614 8877 leader_election.cc:290] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 2 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:04.983542 8952 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.985711 8951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d57fe9ae42243119707a7a47e4dccca" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:04.986398 8951 raft_consensus.cc:3060] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:04.993513 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d57fe9ae42243119707a7a47e4dccca" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:04.974767 8954 raft_consensus.cc:515] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:04.994452 8805 raft_consensus.cc:3055] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:04.995491 8805 raft_consensus.cc:740] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:04.997972 8954 leader_election.cc:290] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 2 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:05.000202 8877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f55b0c696ea74901bb36afdd4d0e338c" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:05.000927 8877 raft_consensus.cc:3055] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:05.001329 8877 raft_consensus.cc:740] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:05.001729 8803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f55b0c696ea74901bb36afdd4d0e338c" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:05.002367 8803 raft_consensus.cc:3060] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:05.002260 8877 consensus_queue.cc:260] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:05.003484 8877 raft_consensus.cc:3060] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:05.008906 8953 raft_consensus.cc:1240] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Rejecting Update request from peer 35eb09ea173d4264b201b528036d5b52 for earlier term 1. Current term is 2. Ops: []
I20260430 07:57:04.996649 8805 consensus_queue.cc:260] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:05.014587 8805 raft_consensus.cc:3060] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:05.016841 8878 raft_consensus.cc:1275] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Refusing update from remote peer 916b9fc1bae040e681e4b82126f3ecad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.021812 9020 consensus_queue.cc:1048] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:05.025542 8700 catalog_manager.cc:5671] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad reported cstate change: term changed from 1 to 2, leader changed from ec08915611e344558a04e8a9c11b6b0a (127.8.119.131) to 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129). New cstate: current_term: 2 leader_uuid: "916b9fc1bae040e681e4b82126f3ecad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:05.054248 8880 raft_consensus.cc:2468] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 2.
I20260430 07:57:05.058068 8804 raft_consensus.cc:2468] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 2.
I20260430 07:57:05.060954 8952 raft_consensus.cc:2468] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 2.
I20260430 07:57:05.064935 8771 leader_election.cc:304] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:05.054467 8953 raft_consensus.cc:1275] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Refusing update from remote peer 916b9fc1bae040e681e4b82126f3ecad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.066115 9020 raft_consensus.cc:2804] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:05.071497 9055 consensus_queue.cc:1048] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:05.071883 9020 raft_consensus.cc:697] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 2 LEADER]: Becoming Leader. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:05.072969 9020 consensus_queue.cc:237] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:05.093657 8702 catalog_manager.cc:5671] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 reported cstate change: term changed from 1 to 2, leader changed from ec08915611e344558a04e8a9c11b6b0a (127.8.119.131) to 35eb09ea173d4264b201b528036d5b52 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "35eb09ea173d4264b201b528036d5b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:05.096370 8702 catalog_manager.cc:5671] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 reported cstate change: term changed from 1 to 2, leader changed from ec08915611e344558a04e8a9c11b6b0a (127.8.119.131) to 35eb09ea173d4264b201b528036d5b52 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "35eb09ea173d4264b201b528036d5b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:05.113698 8700 catalog_manager.cc:5671] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad reported cstate change: term changed from 1 to 2, leader changed from ec08915611e344558a04e8a9c11b6b0a (127.8.119.131) to 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129). New cstate: current_term: 2 leader_uuid: "916b9fc1bae040e681e4b82126f3ecad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
W20260430 07:57:05.158831 8879 consensus_meta.cc:287] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52: Time spent flushing consensus metadata: real 0.622s user 0.015s sys 0.008s
I20260430 07:57:05.158900 8951 raft_consensus.cc:2468] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 2.
I20260430 07:57:05.159586 8879 raft_consensus.cc:515] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:05.160665 8877 raft_consensus.cc:2468] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 2.
I20260430 07:57:05.162190 8879 leader_election.cc:290] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 2 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:05.162422 8803 raft_consensus.cc:2468] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 2.
I20260430 07:57:05.163329 8878 raft_consensus.cc:1240] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Rejecting Update request from peer ec08915611e344558a04e8a9c11b6b0a for earlier term 1. Current term is 2. Ops: []
I20260430 07:57:05.164688 8919 leader_election.cc:304] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:05.166378 9065 raft_consensus.cc:2804] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:05.166927 8803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:05.167455 9038 consensus_queue.cc:1059] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:05.167968 8803 raft_consensus.cc:3060] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:05.168565 9038 raft_consensus.cc:3055] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:05.168923 9038 raft_consensus.cc:740] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:05.166960 9065 raft_consensus.cc:697] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 2 LEADER]: Becoming Leader. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:05.169205 8846 leader_election.cc:304] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:05.169894 9038 consensus_queue.cc:260] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:05.170229 9061 raft_consensus.cc:2804] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:05.170926 9061 raft_consensus.cc:697] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Becoming Leader. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:05.170218 9065 consensus_queue.cc:237] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:05.172077 8951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:05.171766 9061 consensus_queue.cc:237] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:05.173465 9038 raft_consensus.cc:3060] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:05.183971 8701 catalog_manager.cc:5671] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a reported cstate change: term changed from 1 to 2, leader changed from 35eb09ea173d4264b201b528036d5b52 (127.8.119.130) to ec08915611e344558a04e8a9c11b6b0a (127.8.119.131). New cstate: current_term: 2 leader_uuid: "ec08915611e344558a04e8a9c11b6b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:05.190979 8702 catalog_manager.cc:5671] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 reported cstate change: term changed from 1 to 2, leader changed from 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129) to 35eb09ea173d4264b201b528036d5b52 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "35eb09ea173d4264b201b528036d5b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:05.261955 8805 raft_consensus.cc:2468] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 2.
I20260430 07:57:05.318686 8953 raft_consensus.cc:1275] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.320642 9061 consensus_queue.cc:1048] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:05.321323 8805 raft_consensus.cc:1275] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.323004 9061 consensus_queue.cc:1048] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:05.353547 8953 raft_consensus.cc:1275] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.355082 9061 consensus_queue.cc:1048] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:05.361935 8951 raft_consensus.cc:2468] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 2.
I20260430 07:57:05.361691 8803 raft_consensus.cc:2468] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 2.
I20260430 07:57:05.366191 8845 leader_election.cc:304] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
I20260430 07:57:05.367112 9093 raft_consensus.cc:2804] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:05.367530 9093 raft_consensus.cc:697] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Becoming Leader. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:05.368387 9093 consensus_queue.cc:237] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:05.375947 8803 raft_consensus.cc:1275] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.378504 9092 consensus_queue.cc:1048] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:05.387815 8701 catalog_manager.cc:5671] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 reported cstate change: term changed from 1 to 2, leader changed from ec08915611e344558a04e8a9c11b6b0a (127.8.119.131) to 35eb09ea173d4264b201b528036d5b52 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "35eb09ea173d4264b201b528036d5b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:05.629762 8953 raft_consensus.cc:1275] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Refusing update from remote peer 916b9fc1bae040e681e4b82126f3ecad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.631362 9020 consensus_queue.cc:1048] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:05.647997 8878 raft_consensus.cc:1275] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Refusing update from remote peer 916b9fc1bae040e681e4b82126f3ecad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.650499 9017 consensus_queue.cc:1048] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:05.672757 8803 raft_consensus.cc:1275] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.674535 9092 consensus_queue.cc:1048] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:05.685441 8953 raft_consensus.cc:1275] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.687328 9061 consensus_queue.cc:1048] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:05.712057 8803 raft_consensus.cc:1275] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Refusing update from remote peer ec08915611e344558a04e8a9c11b6b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.713371 9038 consensus_queue.cc:1048] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:05.727927 8878 raft_consensus.cc:1275] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Refusing update from remote peer ec08915611e344558a04e8a9c11b6b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.729414 9111 consensus_queue.cc:1048] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260430 07:57:05.852075 8975 debug-util.cc:398] Leaking SignalData structure 0x7b080016e060 after lost signal to thread 8673
W20260430 07:57:05.853122 8975 debug-util.cc:398] Leaking SignalData structure 0x7b080016e740 after lost signal to thread 8736
W20260430 07:57:05.854055 8975 debug-util.cc:398] Leaking SignalData structure 0x7b08001c6a20 after lost signal to thread 8829
W20260430 07:57:05.854954 8975 debug-util.cc:398] Leaking SignalData structure 0x7b080019bbe0 after lost signal to thread 8904
W20260430 07:57:05.855829 8975 debug-util.cc:398] Leaking SignalData structure 0x7b0800092940 after lost signal to thread 8978
I20260430 07:57:05.888727 8803 raft_consensus.cc:1275] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.890287 9092 consensus_queue.cc:1048] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:05.913986 8953 raft_consensus.cc:1275] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:05.919745 9092 consensus_queue.cc:1048] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:07.454967 8670 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20260430 07:57:07.456044 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:07.487983 8803 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320"
dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
mode: GRACEFUL
new_leader_uuid: "ec08915611e344558a04e8a9c11b6b0a"
from {username='slave'} at 127.0.0.1:59216
I20260430 07:57:07.489014 8803 raft_consensus.cc:606] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Received request to transfer leadership to TS ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:07.504954 8878 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
mode: GRACEFUL
new_leader_uuid: "ec08915611e344558a04e8a9c11b6b0a"
from {username='slave'} at 127.0.0.1:53810
I20260430 07:57:07.505591 8878 raft_consensus.cc:606] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Received request to transfer leadership to TS ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:07.506932 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260430 07:57:07.507858 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:07.655436 9114 raft_consensus.cc:993] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad: : Instructing follower ec08915611e344558a04e8a9c11b6b0a to start an election
I20260430 07:57:07.655992 9114 raft_consensus.cc:1081] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Signalling peer ec08915611e344558a04e8a9c11b6b0a to start an election
I20260430 07:57:07.657951 8953 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
from {username='slave'} at 127.0.0.1:59368
I20260430 07:57:07.658705 8953 raft_consensus.cc:493] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:07.660552 8953 raft_consensus.cc:3060] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:07.671605 8953 raft_consensus.cc:515] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:07.673966 8953 leader_election.cc:290] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 2 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:07.675665 8878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:07.675572 8803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:07.676412 8803 raft_consensus.cc:3055] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:07.676728 8803 raft_consensus.cc:740] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:07.676864 8878 raft_consensus.cc:3060] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:07.677640 8803 consensus_queue.cc:260] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:07.678982 8803 raft_consensus.cc:3060] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:07.684237 8878 raft_consensus.cc:2468] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 2.
I20260430 07:57:07.686280 8919 leader_election.cc:304] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:07.686983 8803 raft_consensus.cc:2468] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 2.
I20260430 07:57:07.687208 9111 raft_consensus.cc:2804] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:07.688696 9111 raft_consensus.cc:697] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 2 LEADER]: Becoming Leader. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:07.689782 9111 consensus_queue.cc:237] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:07.698951 8700 catalog_manager.cc:5671] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a reported cstate change: term changed from 1 to 2, leader changed from 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129) to ec08915611e344558a04e8a9c11b6b0a (127.8.119.131). New cstate: current_term: 2 leader_uuid: "ec08915611e344558a04e8a9c11b6b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:07.853922 9092 raft_consensus.cc:993] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52: : Instructing follower ec08915611e344558a04e8a9c11b6b0a to start an election
I20260430 07:57:07.854637 9124 raft_consensus.cc:1081] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Signalling peer ec08915611e344558a04e8a9c11b6b0a to start an election
I20260430 07:57:07.857909 8953 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
from {username='slave'} at 127.0.0.1:59376
I20260430 07:57:07.859014 8953 raft_consensus.cc:493] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:07.859468 8953 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:07.869606 8953 raft_consensus.cc:515] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:07.871737 8953 leader_election.cc:290] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 3 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:07.873307 8803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:07.873229 8878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:07.874099 8803 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:07.874226 8878 raft_consensus.cc:3055] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:07.874648 8878 raft_consensus.cc:740] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:07.875510 8878 consensus_queue.cc:260] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:07.876912 8878 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:07.879642 8803 raft_consensus.cc:2468] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 3.
I20260430 07:57:07.881774 8919 leader_election.cc:304] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:07.882779 9111 raft_consensus.cc:2804] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:07.883272 9111 raft_consensus.cc:697] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 3 LEADER]: Becoming Leader. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:07.884390 9111 consensus_queue.cc:237] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:07.885814 8878 raft_consensus.cc:2468] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 3.
I20260430 07:57:07.897437 8700 catalog_manager.cc:5671] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a reported cstate change: term changed from 2 to 3, leader changed from 35eb09ea173d4264b201b528036d5b52 (127.8.119.130) to ec08915611e344558a04e8a9c11b6b0a (127.8.119.131). New cstate: current_term: 3 leader_uuid: "ec08915611e344558a04e8a9c11b6b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:08.080439 8803 raft_consensus.cc:1275] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Refusing update from remote peer ec08915611e344558a04e8a9c11b6b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:08.082544 9111 consensus_queue.cc:1048] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:08.108245 8878 raft_consensus.cc:1275] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Refusing update from remote peer ec08915611e344558a04e8a9c11b6b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:08.110966 9118 consensus_queue.cc:1048] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:08.313602 8803 raft_consensus.cc:1275] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Refusing update from remote peer ec08915611e344558a04e8a9c11b6b0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:08.315804 9118 consensus_queue.cc:1048] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:57:08.327100 8878 raft_consensus.cc:1275] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Refusing update from remote peer ec08915611e344558a04e8a9c11b6b0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:08.328999 9118 consensus_queue.cc:1048] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:08.509080 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:08.514460 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:08.514894 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:09.516225 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:09.521690 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:09.522166 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:10.523291 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:10.530372 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:10.530882 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:11.531919 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:11.538708 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:11.539198 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:12.540207 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:12.545768 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:12.546449 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:13.547483 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:13.553558 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:13.553995 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:14.603271 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:14.608960 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:14.609745 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20260430 07:57:15.218156 8733 debug-util.cc:398] Leaking SignalData structure 0x7b0800061060 after lost signal to thread 8673
W20260430 07:57:15.218986 8733 debug-util.cc:398] Leaking SignalData structure 0x7b0800061740 after lost signal to thread 8736
W20260430 07:57:15.219772 8733 debug-util.cc:398] Leaking SignalData structure 0x7b08001d3440 after lost signal to thread 8829
W20260430 07:57:15.220434 8733 debug-util.cc:398] Leaking SignalData structure 0x7b08001e4c40 after lost signal to thread 8904
W20260430 07:57:15.221107 8733 debug-util.cc:398] Leaking SignalData structure 0x7b08001a0a80 after lost signal to thread 8978
I20260430 07:57:15.610934 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:15.619935 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:15.620553 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:16.621954 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:16.631433 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:16.632074 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:17.672768 8670 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20260430 07:57:17.679848 8805 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948"
dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
mode: GRACEFUL
new_leader_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:59212
I20260430 07:57:17.681349 8805 raft_consensus.cc:606] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Received request to transfer leadership to TS 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:17.688407 8805 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a"
dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
mode: GRACEFUL
new_leader_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:59212
I20260430 07:57:17.688982 8805 raft_consensus.cc:606] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 2 LEADER]: Received request to transfer leadership to TS 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:17.695362 8805 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8a8897623436482eb062dc3813c3633b"
dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
mode: GRACEFUL
new_leader_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:59212
I20260430 07:57:17.695905 8805 raft_consensus.cc:606] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 2 LEADER]: Received request to transfer leadership to TS 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:17.701573 8951 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
mode: GRACEFUL
new_leader_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:59388
I20260430 07:57:17.702201 8951 raft_consensus.cc:606] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 2 LEADER]: Received request to transfer leadership to TS 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:17.710825 8951 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
mode: GRACEFUL
new_leader_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:59388
I20260430 07:57:17.711387 8951 raft_consensus.cc:606] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 3 LEADER]: Received request to transfer leadership to TS 35eb09ea173d4264b201b528036d5b52
I20260430 07:57:17.779831 9140 raft_consensus.cc:993] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a: : Instructing follower 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:17.780423 9118 raft_consensus.cc:1081] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 2 LEADER]: Signalling peer 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:17.782320 8878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:53790
I20260430 07:57:17.782994 8878 raft_consensus.cc:493] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:17.783484 8878 raft_consensus.cc:3060] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:17.792858 8878 raft_consensus.cc:515] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:17.796818 8878 leader_election.cc:290] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 3 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:17.797763 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:17.797854 8951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ad7efb0efd54c90a593c09fd7e2e320" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:17.798606 8951 raft_consensus.cc:3055] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:17.798907 8951 raft_consensus.cc:740] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:17.799603 8805 raft_consensus.cc:3060] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:17.799741 8951 consensus_queue.cc:260] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:17.800982 8951 raft_consensus.cc:3060] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:17.806432 8951 raft_consensus.cc:2468] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 3.
I20260430 07:57:17.807793 8846 leader_election.cc:304] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:17.808724 9092 raft_consensus.cc:2804] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:17.809530 8805 raft_consensus.cc:2468] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 3.
I20260430 07:57:17.810039 9092 raft_consensus.cc:697] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 3 LEADER]: Becoming Leader. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:17.811024 9092 consensus_queue.cc:237] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:17.823736 8701 catalog_manager.cc:5671] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 reported cstate change: term changed from 2 to 3, leader changed from ec08915611e344558a04e8a9c11b6b0a (127.8.119.131) to 35eb09ea173d4264b201b528036d5b52 (127.8.119.130). New cstate: current_term: 3 leader_uuid: "35eb09ea173d4264b201b528036d5b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:17.871641 9138 raft_consensus.cc:993] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad: : Instructing follower 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:17.872355 9139 raft_consensus.cc:1081] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 2 LEADER]: Signalling peer 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:17.874639 8878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8a8897623436482eb062dc3813c3633b"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:53774
I20260430 07:57:17.875345 8878 raft_consensus.cc:493] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:17.875866 8878 raft_consensus.cc:3060] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:17.883985 8878 raft_consensus.cc:515] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:17.886056 8878 leader_election.cc:290] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 3 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:17.886981 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a8897623436482eb062dc3813c3633b" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:17.887037 8951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a8897623436482eb062dc3813c3633b" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:17.887742 8951 raft_consensus.cc:3060] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:17.887708 8805 raft_consensus.cc:3055] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:17.888259 8805 raft_consensus.cc:740] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:17.889106 8805 consensus_queue.cc:260] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:17.890730 8805 raft_consensus.cc:3060] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:17.893337 8951 raft_consensus.cc:2468] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 3.
I20260430 07:57:17.894405 8846 leader_election.cc:304] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:17.895105 9092 raft_consensus.cc:2804] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:17.895507 9092 raft_consensus.cc:697] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 3 LEADER]: Becoming Leader. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:17.896294 9092 consensus_queue.cc:237] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:17.898471 8805 raft_consensus.cc:2468] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 3.
I20260430 07:57:17.904439 8701 catalog_manager.cc:5671] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 reported cstate change: term changed from 2 to 3, leader changed from 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129) to 35eb09ea173d4264b201b528036d5b52 (127.8.119.130). New cstate: current_term: 3 leader_uuid: "35eb09ea173d4264b201b528036d5b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:17.908982 9138 raft_consensus.cc:993] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad: : Instructing follower 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:17.909561 9139 raft_consensus.cc:1081] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 2 LEADER]: Signalling peer 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:17.910983 8878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:53774
I20260430 07:57:17.911543 8878 raft_consensus.cc:493] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:17.911872 8878 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:17.917347 8878 raft_consensus.cc:515] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:17.918812 8878 leader_election.cc:290] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 3 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:17.919915 8951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:17.920253 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:17.920545 8951 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:17.920908 8805 raft_consensus.cc:3055] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:17.921316 8805 raft_consensus.cc:740] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:17.922314 8805 consensus_queue.cc:260] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:17.924268 8805 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:17.925568 8951 raft_consensus.cc:2468] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 3.
I20260430 07:57:17.926870 8846 leader_election.cc:304] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:17.927618 9092 raft_consensus.cc:2804] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:17.928021 9092 raft_consensus.cc:697] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 3 LEADER]: Becoming Leader. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:17.928751 9092 consensus_queue.cc:237] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:17.932291 8805 raft_consensus.cc:2468] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 3.
I20260430 07:57:17.938511 8701 catalog_manager.cc:5671] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 reported cstate change: term changed from 2 to 3, leader changed from 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129) to 35eb09ea173d4264b201b528036d5b52 (127.8.119.130). New cstate: current_term: 3 leader_uuid: "35eb09ea173d4264b201b528036d5b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:18.010778 9138 raft_consensus.cc:993] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad: : Instructing follower 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:18.011230 9139 raft_consensus.cc:1081] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Signalling peer 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:18.013124 8878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:53774
I20260430 07:57:18.013693 8878 raft_consensus.cc:493] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:18.013970 8878 raft_consensus.cc:3060] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:18.018141 8878 raft_consensus.cc:515] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:18.019645 8878 leader_election.cc:290] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 2 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:18.020895 8951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:18.021064 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:18.021498 8951 raft_consensus.cc:3060] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:18.021641 8805 raft_consensus.cc:3055] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:18.021984 8805 raft_consensus.cc:740] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:18.022720 8805 consensus_queue.cc:260] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:18.023725 8805 raft_consensus.cc:3060] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:18.029888 8805 raft_consensus.cc:2468] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 2.
I20260430 07:57:18.029106 8951 raft_consensus.cc:2468] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 2.
I20260430 07:57:18.031530 8845 leader_election.cc:304] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
I20260430 07:57:18.032967 9092 raft_consensus.cc:2804] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:18.033871 9092 raft_consensus.cc:697] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Becoming Leader. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:18.034834 9092 consensus_queue.cc:237] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:18.043926 8701 catalog_manager.cc:5671] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 reported cstate change: term changed from 1 to 2, leader changed from 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129) to 35eb09ea173d4264b201b528036d5b52 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "35eb09ea173d4264b201b528036d5b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:18.056716 9140 raft_consensus.cc:993] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a: : Instructing follower 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:18.057319 9140 raft_consensus.cc:1081] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 3 LEADER]: Signalling peer 35eb09ea173d4264b201b528036d5b52 to start an election
I20260430 07:57:18.059365 8878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
from {username='slave'} at 127.0.0.1:53790
I20260430 07:57:18.059868 8878 raft_consensus.cc:493] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:18.060135 8878 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:18.066865 8878 raft_consensus.cc:515] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:18.071005 8878 leader_election.cc:290] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 4 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:18.072193 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:18.072207 8951 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "35eb09ea173d4264b201b528036d5b52" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:18.072769 8805 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:18.072938 8951 raft_consensus.cc:3055] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:57:18.073334 8951 raft_consensus.cc:740] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:18.074090 8951 consensus_queue.cc:260] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:18.075075 8951 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:18.077337 8805 raft_consensus.cc:2468] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 4.
I20260430 07:57:18.078943 8845 leader_election.cc:304] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
I20260430 07:57:18.079809 9092 raft_consensus.cc:2804] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:57:18.080284 9092 raft_consensus.cc:697] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 4 LEADER]: Becoming Leader. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:18.080698 8951 raft_consensus.cc:2468] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35eb09ea173d4264b201b528036d5b52 in term 4.
I20260430 07:57:18.081123 9092 consensus_queue.cc:237] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:18.090548 8701 catalog_manager.cc:5671] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 reported cstate change: term changed from 3 to 4, leader changed from ec08915611e344558a04e8a9c11b6b0a (127.8.119.131) to 35eb09ea173d4264b201b528036d5b52 (127.8.119.130). New cstate: current_term: 4 leader_uuid: "35eb09ea173d4264b201b528036d5b52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:18.232247 8951 raft_consensus.cc:1275] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:18.233922 9092 consensus_queue.cc:1048] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:18.244879 8805 raft_consensus.cc:1275] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:18.246754 9141 consensus_queue.cc:1048] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:57:18.291283 8805 raft_consensus.cc:1275] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:18.292513 9141 consensus_queue.cc:1048] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:18.302177 8951 raft_consensus.cc:1275] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:18.305820 9157 consensus_queue.cc:1048] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260430 07:57:18.439620 8951 raft_consensus.cc:1275] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:18.440832 9141 consensus_queue.cc:1048] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:18.449771 8805 raft_consensus.cc:1275] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:18.451258 9157 consensus_queue.cc:1048] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:18.468338 8805 raft_consensus.cc:1275] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 4 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260430 07:57:18.469547 9157 consensus_queue.cc:1048] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:57:18.480078 8951 raft_consensus.cc:1275] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 4 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260430 07:57:18.481630 9141 consensus_queue.cc:1048] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:57:18.547428 8805 raft_consensus.cc:1275] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:18.548610 9141 consensus_queue.cc:1048] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:18.556938 8951 raft_consensus.cc:1275] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Refusing update from remote peer 35eb09ea173d4264b201b528036d5b52: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:18.558208 9092 consensus_queue.cc:1048] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260430 07:57:20.784761 8733 debug-util.cc:398] Leaking SignalData structure 0x7b080017b480 after lost signal to thread 8673
W20260430 07:57:20.785953 8733 debug-util.cc:398] Leaking SignalData structure 0x7b08001e4a40 after lost signal to thread 8736
W20260430 07:57:20.786808 8733 debug-util.cc:398] Leaking SignalData structure 0x7b0800061180 after lost signal to thread 8829
W20260430 07:57:20.787839 8733 debug-util.cc:398] Leaking SignalData structure 0x7b08001a7460 after lost signal to thread 8904
W20260430 07:57:20.788745 8733 debug-util.cc:398] Leaking SignalData structure 0x7b08001ff980 after lost signal to thread 8978
I20260430 07:57:20.790196 8670 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20260430 07:57:20.791033 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:20.798781 8878 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
mode: GRACEFUL
new_leader_uuid: "ec08915611e344558a04e8a9c11b6b0a"
from {username='slave'} at 127.0.0.1:53810
I20260430 07:57:20.799612 8878 raft_consensus.cc:606] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Received request to transfer leadership to TS ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:20.804483 8878 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
mode: GRACEFUL
new_leader_uuid: "916b9fc1bae040e681e4b82126f3ecad"
from {username='slave'} at 127.0.0.1:53810
I20260430 07:57:20.805114 8878 raft_consensus.cc:606] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 4 LEADER]: Received request to transfer leadership to TS 916b9fc1bae040e681e4b82126f3ecad
I20260430 07:57:20.808293 8878 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
mode: GRACEFUL
new_leader_uuid: "ec08915611e344558a04e8a9c11b6b0a"
from {username='slave'} at 127.0.0.1:53810
I20260430 07:57:20.808768 8878 raft_consensus.cc:606] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 3 LEADER]: Received request to transfer leadership to TS ec08915611e344558a04e8a9c11b6b0a
I20260430 07:57:20.813797 8878 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
mode: GRACEFUL
new_leader_uuid: "916b9fc1bae040e681e4b82126f3ecad"
from {username='slave'} at 127.0.0.1:53810
I20260430 07:57:20.814332 8878 raft_consensus.cc:606] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Received request to transfer leadership to TS 916b9fc1bae040e681e4b82126f3ecad
I20260430 07:57:20.816694 8878 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79"
dest_uuid: "35eb09ea173d4264b201b528036d5b52"
mode: GRACEFUL
new_leader_uuid: "916b9fc1bae040e681e4b82126f3ecad"
from {username='slave'} at 127.0.0.1:53810
I20260430 07:57:20.817147 8878 raft_consensus.cc:606] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Received request to transfer leadership to TS 916b9fc1bae040e681e4b82126f3ecad
I20260430 07:57:20.818495 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260430 07:57:20.818989 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:21.016544 9171 raft_consensus.cc:993] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52: : Instructing follower ec08915611e344558a04e8a9c11b6b0a to start an election
I20260430 07:57:21.017086 9157 raft_consensus.cc:1081] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 3 LEADER]: Signalling peer ec08915611e344558a04e8a9c11b6b0a to start an election
I20260430 07:57:21.018738 8953 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
from {username='slave'} at 127.0.0.1:59376
I20260430 07:57:21.019331 8953 raft_consensus.cc:493] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:21.019667 8953 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:21.024353 8953 raft_consensus.cc:515] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.026297 8953 leader_election.cc:290] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 4 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:21.027576 8878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:21.027374 8803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a14c736da5b465d8a0f8a7971a2840a" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:21.028208 8878 raft_consensus.cc:3055] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:57:21.028342 8803 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:21.028517 8878 raft_consensus.cc:740] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:21.029325 8878 consensus_queue.cc:260] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.030676 8878 raft_consensus.cc:3060] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:21.031108 9171 raft_consensus.cc:993] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52: : Instructing follower 916b9fc1bae040e681e4b82126f3ecad to start an election
I20260430 07:57:21.031603 9171 raft_consensus.cc:1081] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Signalling peer 916b9fc1bae040e681e4b82126f3ecad to start an election
I20260430 07:57:21.033401 8805 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79"
dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
from {username='slave'} at 127.0.0.1:59202
I20260430 07:57:21.033841 8805 raft_consensus.cc:493] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:21.034121 8805 raft_consensus.cc:3060] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:21.035236 8803 raft_consensus.cc:2468] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 4.
I20260430 07:57:21.036608 8919 leader_election.cc:304] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:21.037673 9170 raft_consensus.cc:2804] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:57:21.037878 8878 raft_consensus.cc:2468] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 4.
I20260430 07:57:21.039007 9170 raft_consensus.cc:697] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 4 LEADER]: Becoming Leader. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:21.039542 8805 raft_consensus.cc:515] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.040565 9170 consensus_queue.cc:237] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.041900 8805 leader_election.cc:290] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 3 election: Requested vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:21.043318 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:21.043625 8878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8a431f0e0bb4895aa5627ebc3ae1d79" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:21.043936 8953 raft_consensus.cc:3060] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:21.044214 8878 raft_consensus.cc:3055] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:21.044577 8878 raft_consensus.cc:740] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:21.045166 8878 consensus_queue.cc:260] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.046195 8878 raft_consensus.cc:3060] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:21.050690 8878 raft_consensus.cc:2468] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 3.
I20260430 07:57:21.052340 8770 leader_election.cc:304] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
I20260430 07:57:21.053915 8953 raft_consensus.cc:2468] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 3.
I20260430 07:57:21.057168 9174 raft_consensus.cc:2804] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:21.059603 9174 raft_consensus.cc:697] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 3 LEADER]: Becoming Leader. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:21.060879 9174 consensus_queue.cc:237] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.062119 8700 catalog_manager.cc:5671] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a reported cstate change: term changed from 3 to 4, leader changed from 35eb09ea173d4264b201b528036d5b52 (127.8.119.130) to ec08915611e344558a04e8a9c11b6b0a (127.8.119.131). New cstate: current_term: 4 leader_uuid: "ec08915611e344558a04e8a9c11b6b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:21.068516 8701 catalog_manager.cc:5671] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad reported cstate change: term changed from 2 to 3, leader changed from 35eb09ea173d4264b201b528036d5b52 (127.8.119.130) to 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129). New cstate: current_term: 3 leader_uuid: "916b9fc1bae040e681e4b82126f3ecad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:21.166709 9171 raft_consensus.cc:993] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52: : Instructing follower 916b9fc1bae040e681e4b82126f3ecad to start an election
I20260430 07:57:21.167232 9171 raft_consensus.cc:1081] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Signalling peer 916b9fc1bae040e681e4b82126f3ecad to start an election
I20260430 07:57:21.168987 8805 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364"
dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
from {username='slave'} at 127.0.0.1:59202
I20260430 07:57:21.169482 8805 raft_consensus.cc:493] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:21.169716 8805 raft_consensus.cc:3060] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:21.174742 8805 raft_consensus.cc:515] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.176353 8805 leader_election.cc:290] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 3 election: Requested vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:21.177481 8878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:21.177642 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "936f73b154f54a41bfc33e3cfac79364" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:21.178015 8878 raft_consensus.cc:3055] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:21.178402 8953 raft_consensus.cc:3060] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:21.178411 8878 raft_consensus.cc:740] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:21.179265 8878 consensus_queue.cc:260] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.180200 8878 raft_consensus.cc:3060] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:21.183918 8953 raft_consensus.cc:2468] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 3.
I20260430 07:57:21.185128 8771 leader_election.cc:304] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916b9fc1bae040e681e4b82126f3ecad, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:21.185631 8878 raft_consensus.cc:2468] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 3.
I20260430 07:57:21.186407 9157 raft_consensus.cc:993] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52: : Instructing follower 916b9fc1bae040e681e4b82126f3ecad to start an election
I20260430 07:57:21.185940 9174 raft_consensus.cc:2804] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:21.186987 9171 raft_consensus.cc:1081] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 4 LEADER]: Signalling peer 916b9fc1bae040e681e4b82126f3ecad to start an election
I20260430 07:57:21.187085 9174 raft_consensus.cc:697] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 3 LEADER]: Becoming Leader. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:21.188216 9174 consensus_queue.cc:237] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.189702 8805 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d"
dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
from {username='slave'} at 127.0.0.1:59202
I20260430 07:57:21.190258 8805 raft_consensus.cc:493] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:21.190554 8805 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:57:21.196409 8805 raft_consensus.cc:515] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.196611 8701 catalog_manager.cc:5671] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad reported cstate change: term changed from 2 to 3, leader changed from 35eb09ea173d4264b201b528036d5b52 (127.8.119.130) to 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129). New cstate: current_term: 3 leader_uuid: "916b9fc1bae040e681e4b82126f3ecad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:21.198490 8805 leader_election.cc:290] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 5 election: Requested vote from peers 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379), ec08915611e344558a04e8a9c11b6b0a (127.8.119.131:42455)
I20260430 07:57:21.198792 8878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:21.199383 8878 raft_consensus.cc:3055] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:57:21.199710 8878 raft_consensus.cc:740] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:21.200402 8878 consensus_queue.cc:260] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.201905 8878 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:57:21.202419 8953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a39a3eef27b4abaab420cdcbd358e2d" candidate_uuid: "916b9fc1bae040e681e4b82126f3ecad" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
I20260430 07:57:21.202975 8953 raft_consensus.cc:3060] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:57:21.209839 8878 raft_consensus.cc:2468] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 5.
I20260430 07:57:21.210325 8953 raft_consensus.cc:2468] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916b9fc1bae040e681e4b82126f3ecad in term 5.
I20260430 07:57:21.211097 8770 leader_election.cc:304] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, 916b9fc1bae040e681e4b82126f3ecad; no voters:
I20260430 07:57:21.212399 9174 raft_consensus.cc:2804] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:57:21.212880 9174 raft_consensus.cc:697] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 5 LEADER]: Becoming Leader. State: Replica: 916b9fc1bae040e681e4b82126f3ecad, State: Running, Role: LEADER
I20260430 07:57:21.213742 9174 consensus_queue.cc:237] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } }
I20260430 07:57:21.222872 8701 catalog_manager.cc:5671] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad reported cstate change: term changed from 4 to 5, leader changed from 35eb09ea173d4264b201b528036d5b52 (127.8.119.130) to 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129). New cstate: current_term: 5 leader_uuid: "916b9fc1bae040e681e4b82126f3ecad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:21.315279 9157 raft_consensus.cc:993] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52: : Instructing follower ec08915611e344558a04e8a9c11b6b0a to start an election
I20260430 07:57:21.315716 9171 raft_consensus.cc:1081] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Signalling peer ec08915611e344558a04e8a9c11b6b0a to start an election
I20260430 07:57:21.317916 8953 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948"
dest_uuid: "ec08915611e344558a04e8a9c11b6b0a"
from {username='slave'} at 127.0.0.1:59376
I20260430 07:57:21.318399 8953 raft_consensus.cc:493] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:21.318758 8953 raft_consensus.cc:3060] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:21.323436 8953 raft_consensus.cc:515] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:21.325335 8953 leader_election.cc:290] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 3 election: Requested vote from peers 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251), 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379)
I20260430 07:57:21.326448 8878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "35eb09ea173d4264b201b528036d5b52"
I20260430 07:57:21.327082 8878 raft_consensus.cc:3055] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:21.327340 8878 raft_consensus.cc:740] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 35eb09ea173d4264b201b528036d5b52, State: Running, Role: LEADER
I20260430 07:57:21.327755 8805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c58740f0c24968927cea27ad0d2948" candidate_uuid: "ec08915611e344558a04e8a9c11b6b0a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "916b9fc1bae040e681e4b82126f3ecad"
I20260430 07:57:21.328276 8878 consensus_queue.cc:260] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:21.328670 8805 raft_consensus.cc:3060] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:21.329416 8878 raft_consensus.cc:3060] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:21.336328 8805 raft_consensus.cc:2468] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 3.
I20260430 07:57:21.336426 8878 raft_consensus.cc:2468] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec08915611e344558a04e8a9c11b6b0a in term 3.
I20260430 07:57:21.337510 8919 leader_election.cc:304] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35eb09ea173d4264b201b528036d5b52, ec08915611e344558a04e8a9c11b6b0a; no voters:
I20260430 07:57:21.338693 9170 raft_consensus.cc:2804] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:21.339768 9170 raft_consensus.cc:697] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 3 LEADER]: Becoming Leader. State: Replica: ec08915611e344558a04e8a9c11b6b0a, State: Running, Role: LEADER
I20260430 07:57:21.340935 9170 consensus_queue.cc:237] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } }
I20260430 07:57:21.350328 8701 catalog_manager.cc:5671] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a reported cstate change: term changed from 2 to 3, leader changed from 35eb09ea173d4264b201b528036d5b52 (127.8.119.130) to ec08915611e344558a04e8a9c11b6b0a (127.8.119.131). New cstate: current_term: 3 leader_uuid: "ec08915611e344558a04e8a9c11b6b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:21.435374 8805 raft_consensus.cc:1275] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 4 FOLLOWER]: Refusing update from remote peer ec08915611e344558a04e8a9c11b6b0a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260430 07:57:21.436859 9170 consensus_queue.cc:1048] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:57:21.449865 8878 raft_consensus.cc:1275] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 4 FOLLOWER]: Refusing update from remote peer ec08915611e344558a04e8a9c11b6b0a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260430 07:57:21.451568 9187 consensus_queue.cc:1048] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:57:21.523191 8953 raft_consensus.cc:1275] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Refusing update from remote peer 916b9fc1bae040e681e4b82126f3ecad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:21.524981 9174 consensus_queue.cc:1048] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:21.536279 8878 raft_consensus.cc:1275] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Refusing update from remote peer 916b9fc1bae040e681e4b82126f3ecad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:21.538275 9174 consensus_queue.cc:1048] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:57:21.591286 8878 raft_consensus.cc:1275] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Refusing update from remote peer 916b9fc1bae040e681e4b82126f3ecad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:21.592640 9174 consensus_queue.cc:1048] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:21.598934 8953 raft_consensus.cc:1275] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Refusing update from remote peer 916b9fc1bae040e681e4b82126f3ecad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:21.600409 9174 consensus_queue.cc:1048] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:21.666462 8953 raft_consensus.cc:1275] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 5 FOLLOWER]: Refusing update from remote peer 916b9fc1bae040e681e4b82126f3ecad: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260430 07:57:21.668597 9174 consensus_queue.cc:1048] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec08915611e344558a04e8a9c11b6b0a" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 42455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:57:21.687328 8878 raft_consensus.cc:1275] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 5 FOLLOWER]: Refusing update from remote peer 916b9fc1bae040e681e4b82126f3ecad: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260430 07:57:21.690605 9174 consensus_queue.cc:1048] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:57:21.761847 8805 raft_consensus.cc:1275] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Refusing update from remote peer ec08915611e344558a04e8a9c11b6b0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:21.763193 9187 consensus_queue.cc:1048] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "916b9fc1bae040e681e4b82126f3ecad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:21.774709 8878 raft_consensus.cc:1275] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Refusing update from remote peer ec08915611e344558a04e8a9c11b6b0a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:21.776710 9187 consensus_queue.cc:1048] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35eb09ea173d4264b201b528036d5b52" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:57:21.820379 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:21.826031 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:21.826447 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:22.827447 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:22.832706 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:22.833155 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:23.834116 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:23.839150 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:23.839617 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:24.840615 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:24.846838 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:24.847244 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:25.848632 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:25.856004 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:25.856616 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:26.857858 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:26.863617 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:26.864154 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:27.865597 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:27.871872 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:27.872344 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:28.873821 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:28.880623 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:28.881187 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:29.882794 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:29.890398 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:29.890933 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:30.930930 8670 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20260430 07:57:30.931478 8670 tablet_server.cc:179] TabletServer@127.8.119.129:0 shutting down...
W20260430 07:57:30.948006 8845 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260430 07:57:30.956521 8845 consensus_peers.cc:597] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 -> Peer 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251): Couldn't send request to peer 916b9fc1bae040e681e4b82126f3ecad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260430 07:57:30.977142 8845 consensus_peers.cc:597] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 -> Peer 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251): Couldn't send request to peer 916b9fc1bae040e681e4b82126f3ecad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260430 07:57:30.977807 8919 consensus_peers.cc:597] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a -> Peer 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251): Couldn't send request to peer 916b9fc1bae040e681e4b82126f3ecad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:57:30.990368 8670 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260430 07:57:30.991742 8670 tablet_replica.cc:333] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad: stopping tablet replica
I20260430 07:57:30.993031 8670 raft_consensus.cc:2243] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 5 LEADER]: Raft consensus shutting down.
I20260430 07:57:30.995018 8670 raft_consensus.cc:2272] T 3a39a3eef27b4abaab420cdcbd358e2d P 916b9fc1bae040e681e4b82126f3ecad [term 5 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:30.997841 8670 tablet_replica.cc:333] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad: stopping tablet replica
I20260430 07:57:30.998415 8670 raft_consensus.cc:2243] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:30.998994 8670 raft_consensus.cc:2272] T 37c58740f0c24968927cea27ad0d2948 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.000898 8670 tablet_replica.cc:333] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad: stopping tablet replica
I20260430 07:57:31.001693 8670 raft_consensus.cc:2243] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.002156 8670 raft_consensus.cc:2272] T 8a8897623436482eb062dc3813c3633b P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.004856 8670 tablet_replica.cc:333] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad: stopping tablet replica
I20260430 07:57:31.005554 8670 raft_consensus.cc:2243] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 3 LEADER]: Raft consensus shutting down.
W20260430 07:57:31.005800 8845 consensus_peers.cc:597] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 -> Peer 916b9fc1bae040e681e4b82126f3ecad (127.8.119.129:42251): Couldn't send request to peer 916b9fc1bae040e681e4b82126f3ecad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:57:31.006685 8670 raft_consensus.cc:2272] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.008464 8670 tablet_replica.cc:333] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad: stopping tablet replica
I20260430 07:57:31.008940 8670 raft_consensus.cc:2243] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.009438 8670 raft_consensus.cc:2272] T f55b0c696ea74901bb36afdd4d0e338c P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.011149 8670 tablet_replica.cc:333] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad: stopping tablet replica
I20260430 07:57:31.011600 8670 raft_consensus.cc:2243] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 3 LEADER]: Raft consensus shutting down.
I20260430 07:57:31.012656 8670 raft_consensus.cc:2272] T 936f73b154f54a41bfc33e3cfac79364 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.014530 8670 tablet_replica.cc:333] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad: stopping tablet replica
I20260430 07:57:31.015110 8670 raft_consensus.cc:2243] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 4 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.015512 8670 raft_consensus.cc:2272] T 6a14c736da5b465d8a0f8a7971a2840a P 916b9fc1bae040e681e4b82126f3ecad [term 4 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.017234 8670 tablet_replica.cc:333] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad: stopping tablet replica
I20260430 07:57:31.017771 8670 raft_consensus.cc:2243] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.018180 8670 raft_consensus.cc:2272] T 1ad7efb0efd54c90a593c09fd7e2e320 P 916b9fc1bae040e681e4b82126f3ecad [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.019874 8670 tablet_replica.cc:333] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad: stopping tablet replica
I20260430 07:57:31.020278 8670 raft_consensus.cc:2243] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.020656 8670 raft_consensus.cc:2272] T 6d57fe9ae42243119707a7a47e4dccca P 916b9fc1bae040e681e4b82126f3ecad [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.137846 8670 tablet_server.cc:196] TabletServer@127.8.119.129:0 shutdown complete.
I20260430 07:57:31.162496 8670 tablet_server.cc:179] TabletServer@127.8.119.130:0 shutting down...
I20260430 07:57:31.204514 8670 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260430 07:57:31.205196 8670 tablet_replica.cc:333] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52: stopping tablet replica
I20260430 07:57:31.205904 8670 raft_consensus.cc:2243] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.206471 8670 raft_consensus.cc:2272] T f55b0c696ea74901bb36afdd4d0e338c P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.208423 8670 tablet_replica.cc:333] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52: stopping tablet replica
I20260430 07:57:31.208966 8670 raft_consensus.cc:2243] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 2 LEADER]: Raft consensus shutting down.
I20260430 07:57:31.209834 8670 raft_consensus.cc:2272] T 6d57fe9ae42243119707a7a47e4dccca P 35eb09ea173d4264b201b528036d5b52 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.211660 8670 tablet_replica.cc:333] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52: stopping tablet replica
I20260430 07:57:31.212206 8670 raft_consensus.cc:2243] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.212868 8670 raft_consensus.cc:2272] T 3a39a3eef27b4abaab420cdcbd358e2d P 35eb09ea173d4264b201b528036d5b52 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.216323 8670 tablet_replica.cc:333] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52: stopping tablet replica
I20260430 07:57:31.217082 8670 raft_consensus.cc:2243] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.217733 8670 raft_consensus.cc:2272] T 936f73b154f54a41bfc33e3cfac79364 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.219363 8670 tablet_replica.cc:333] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52: stopping tablet replica
I20260430 07:57:31.219815 8670 raft_consensus.cc:2243] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.220180 8670 raft_consensus.cc:2272] T 6a14c736da5b465d8a0f8a7971a2840a P 35eb09ea173d4264b201b528036d5b52 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.221947 8670 tablet_replica.cc:333] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52: stopping tablet replica
I20260430 07:57:31.222404 8670 raft_consensus.cc:2243] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 3 LEADER]: Raft consensus shutting down.
I20260430 07:57:31.223192 8670 raft_consensus.cc:2272] T 1ad7efb0efd54c90a593c09fd7e2e320 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.224933 8670 tablet_replica.cc:333] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52: stopping tablet replica
I20260430 07:57:31.225530 8670 raft_consensus.cc:2243] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.225901 8670 raft_consensus.cc:2272] T d8a431f0e0bb4895aa5627ebc3ae1d79 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.227541 8670 tablet_replica.cc:333] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52: stopping tablet replica
I20260430 07:57:31.227965 8670 raft_consensus.cc:2243] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 3 LEADER]: Raft consensus shutting down.
I20260430 07:57:31.228699 8670 raft_consensus.cc:2272] T 8a8897623436482eb062dc3813c3633b P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.230875 8670 tablet_replica.cc:333] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52: stopping tablet replica
I20260430 07:57:31.231362 8670 raft_consensus.cc:2243] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.231844 8670 raft_consensus.cc:2272] T 37c58740f0c24968927cea27ad0d2948 P 35eb09ea173d4264b201b528036d5b52 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260430 07:57:31.233503 8919 consensus_peers.cc:597] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a -> Peer 35eb09ea173d4264b201b528036d5b52 (127.8.119.130:39379): Couldn't send request to peer 35eb09ea173d4264b201b528036d5b52. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:57:31.264264 8670 tablet_server.cc:196] TabletServer@127.8.119.130:0 shutdown complete.
I20260430 07:57:31.286171 8670 tablet_server.cc:179] TabletServer@127.8.119.131:0 shutting down...
I20260430 07:57:31.322419 8670 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260430 07:57:31.323278 8670 tablet_replica.cc:333] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a: stopping tablet replica
I20260430 07:57:31.324118 8670 raft_consensus.cc:2243] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 2 LEADER]: Raft consensus shutting down.
I20260430 07:57:31.325743 8670 raft_consensus.cc:2272] T f55b0c696ea74901bb36afdd4d0e338c P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.328658 8670 tablet_replica.cc:333] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a: stopping tablet replica
I20260430 07:57:31.329396 8670 raft_consensus.cc:2243] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 5 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.329906 8670 raft_consensus.cc:2272] T 3a39a3eef27b4abaab420cdcbd358e2d P ec08915611e344558a04e8a9c11b6b0a [term 5 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.332316 8670 tablet_replica.cc:333] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a: stopping tablet replica
I20260430 07:57:31.332996 8670 raft_consensus.cc:2243] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 3 LEADER]: Raft consensus shutting down.
I20260430 07:57:31.333865 8670 raft_consensus.cc:2272] T 37c58740f0c24968927cea27ad0d2948 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.335866 8670 tablet_replica.cc:333] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a: stopping tablet replica
I20260430 07:57:31.336438 8670 raft_consensus.cc:2243] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 4 LEADER]: Raft consensus shutting down.
I20260430 07:57:31.337385 8670 raft_consensus.cc:2272] T 6a14c736da5b465d8a0f8a7971a2840a P ec08915611e344558a04e8a9c11b6b0a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.339434 8670 tablet_replica.cc:333] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a: stopping tablet replica
I20260430 07:57:31.340020 8670 raft_consensus.cc:2243] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.340523 8670 raft_consensus.cc:2272] T 936f73b154f54a41bfc33e3cfac79364 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.342339 8670 tablet_replica.cc:333] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a: stopping tablet replica
I20260430 07:57:31.342854 8670 raft_consensus.cc:2243] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.343351 8670 raft_consensus.cc:2272] T d8a431f0e0bb4895aa5627ebc3ae1d79 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.346417 8670 tablet_replica.cc:333] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a: stopping tablet replica
I20260430 07:57:31.347003 8670 raft_consensus.cc:2243] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.348253 8670 raft_consensus.cc:2272] T 8a8897623436482eb062dc3813c3633b P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.350499 8670 tablet_replica.cc:333] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a: stopping tablet replica
I20260430 07:57:31.351011 8670 raft_consensus.cc:2243] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.351459 8670 raft_consensus.cc:2272] T 6d57fe9ae42243119707a7a47e4dccca P ec08915611e344558a04e8a9c11b6b0a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.353420 8670 tablet_replica.cc:333] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a: stopping tablet replica
I20260430 07:57:31.353910 8670 raft_consensus.cc:2243] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:57:31.354622 8670 raft_consensus.cc:2272] T 1ad7efb0efd54c90a593c09fd7e2e320 P ec08915611e344558a04e8a9c11b6b0a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.422169 8670 tablet_server.cc:196] TabletServer@127.8.119.131:0 shutdown complete.
I20260430 07:57:31.443044 8670 master.cc:562] Master@127.8.119.190:35737 shutting down...
I20260430 07:57:31.694861 8670 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 1 LEADER]: Raft consensus shutting down.
I20260430 07:57:31.695508 8670 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:57:31.695876 8670 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0ac3f16b3d5b4753b196ac832dad0046: stopping tablet replica
I20260430 07:57:31.769032 8670 master.cc:584] Master@127.8.119.190:35737 shutdown complete.
35eb09ea173d4264b201b528036d5b52 4
916b9fc1bae040e681e4b82126f3ecad 4
ec08915611e344558a04e8a9c11b6b0a 1
35eb09ea173d4264b201b528036d5b52 3
916b9fc1bae040e681e4b82126f3ecad 3
ec08915611e344558a04e8a9c11b6b0a 3
35eb09ea173d4264b201b528036d5b52 8
916b9fc1bae040e681e4b82126f3ecad 0
ec08915611e344558a04e8a9c11b6b0a 1
35eb09ea173d4264b201b528036d5b52 3
916b9fc1bae040e681e4b82126f3ecad 3
ec08915611e344558a04e8a9c11b6b0a 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (32938 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260430 07:57:31.831262 8670 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.119.190:46599
I20260430 07:57:31.833004 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:57:31.840883 9219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:31.841822 9220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:31.842919 9222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:57:31.844898 8670 server_base.cc:1061] running on GCE node
I20260430 07:57:31.848532 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:57:31.849917 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:57:31.851155 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535851850873 us; error 0 us; skew 500 ppm
I20260430 07:57:31.852284 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:57:31.856088 8670 webserver.cc:492] Webserver started at http://127.8.119.190:40477/ using document root <none> and password file <none>
I20260430 07:57:31.856703 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:57:31.856880 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:57:31.857177 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:57:31.858484 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/master-0-root/instance:
uuid: "1df1c13247cb4d6eb41c854c76db4fb2"
format_stamp: "Formatted at 2026-04-30 07:57:31 on dist-test-slave-6ct4"
I20260430 07:57:31.863626 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 07:57:31.870255 9227 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:31.871600 8670 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260430 07:57:31.871953 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/master-0-root
uuid: "1df1c13247cb4d6eb41c854c76db4fb2"
format_stamp: "Formatted at 2026-04-30 07:57:31 on dist-test-slave-6ct4"
I20260430 07:57:31.872285 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:57:31.913235 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:57:31.914878 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:57:31.965783 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.190:46599
I20260430 07:57:31.965956 9278 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.190:46599 every 8 connection(s)
I20260430 07:57:31.970295 9279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:31.983790 9279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2: Bootstrap starting.
I20260430 07:57:31.989368 9279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:31.994222 9279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2: No bootstrap required, opened a new log
I20260430 07:57:31.997471 9279 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1c13247cb4d6eb41c854c76db4fb2" member_type: VOTER }
I20260430 07:57:31.997944 9279 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:31.998231 9279 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df1c13247cb4d6eb41c854c76db4fb2, State: Initialized, Role: FOLLOWER
I20260430 07:57:31.998829 9279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1c13247cb4d6eb41c854c76db4fb2" member_type: VOTER }
I20260430 07:57:31.999361 9279 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 07:57:31.999599 9279 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 07:57:31.999924 9279 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:32.004459 9279 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1c13247cb4d6eb41c854c76db4fb2" member_type: VOTER }
I20260430 07:57:32.005070 9279 leader_election.cc:304] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1df1c13247cb4d6eb41c854c76db4fb2; no voters:
I20260430 07:57:32.006279 9279 leader_election.cc:290] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 07:57:32.006688 9282 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:32.007972 9282 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 1df1c13247cb4d6eb41c854c76db4fb2, State: Running, Role: LEADER
I20260430 07:57:32.008704 9282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1c13247cb4d6eb41c854c76db4fb2" member_type: VOTER }
I20260430 07:57:32.009142 9279 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260430 07:57:32.015213 9284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1df1c13247cb4d6eb41c854c76db4fb2. Latest consensus state: current_term: 1 leader_uuid: "1df1c13247cb4d6eb41c854c76db4fb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1c13247cb4d6eb41c854c76db4fb2" member_type: VOTER } }
I20260430 07:57:32.016351 9284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [sys.catalog]: This master's current role is: LEADER
I20260430 07:57:32.015377 9283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1df1c13247cb4d6eb41c854c76db4fb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1c13247cb4d6eb41c854c76db4fb2" member_type: VOTER } }
I20260430 07:57:32.017776 9283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [sys.catalog]: This master's current role is: LEADER
I20260430 07:57:32.022264 9290 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 07:57:32.027827 9290 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 07:57:32.029706 8670 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260430 07:57:32.039117 9290 catalog_manager.cc:1357] Generated new cluster ID: a6b962f0e5f04e4cb3c5bfaefdc6bec1
I20260430 07:57:32.039439 9290 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 07:57:32.064666 9290 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 07:57:32.065953 9290 catalog_manager.cc:1514] Loading token signing keys...
I20260430 07:57:32.083811 9290 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2: Generated new TSK 0
I20260430 07:57:32.084468 9290 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 07:57:32.097162 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:57:32.104142 9300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:32.105237 9301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:32.107184 9303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:57:32.108701 8670 server_base.cc:1061] running on GCE node
I20260430 07:57:32.109835 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:57:32.110163 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:57:32.110316 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535852110299 us; error 0 us; skew 500 ppm
I20260430 07:57:32.110939 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:57:32.113672 8670 webserver.cc:492] Webserver started at http://127.8.119.129:45909/ using document root <none> and password file <none>
I20260430 07:57:32.114187 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:57:32.114409 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:57:32.114686 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:57:32.115792 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-0-root/instance:
uuid: "48cc0d3c7a74492b9f815c91740b38ad"
format_stamp: "Formatted at 2026-04-30 07:57:32 on dist-test-slave-6ct4"
I20260430 07:57:32.122196 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260430 07:57:32.126961 9308 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:32.128412 8670 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260430 07:57:32.128762 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-0-root
uuid: "48cc0d3c7a74492b9f815c91740b38ad"
format_stamp: "Formatted at 2026-04-30 07:57:32 on dist-test-slave-6ct4"
I20260430 07:57:32.129098 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:57:32.146260 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:57:32.147526 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:57:32.149082 8670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 07:57:32.151551 8670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 07:57:32.151903 8670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:32.152177 8670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 07:57:32.152371 8670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:32.202435 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.129:34037
I20260430 07:57:32.202605 9370 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.129:34037 every 8 connection(s)
I20260430 07:57:32.207782 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:57:32.217828 9375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:32.218991 9376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:32.220960 9378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:57:32.222203 8670 server_base.cc:1061] running on GCE node
I20260430 07:57:32.224171 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:57:32.224457 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:57:32.224702 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535852224676 us; error 0 us; skew 500 ppm
I20260430 07:57:32.225391 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:57:32.225396 9371 heartbeater.cc:344] Connected to a master server at 127.8.119.190:46599
I20260430 07:57:32.226002 9371 heartbeater.cc:461] Registering TS with master...
I20260430 07:57:32.226716 9371 heartbeater.cc:507] Master 127.8.119.190:46599 requested a full tablet report, sending...
I20260430 07:57:32.229568 8670 webserver.cc:492] Webserver started at http://127.8.119.130:41145/ using document root <none> and password file <none>
I20260430 07:57:32.229668 9244 ts_manager.cc:194] Registered new tserver with Master: 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037)
I20260430 07:57:32.230284 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:57:32.230522 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:57:32.230801 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:57:32.231781 9244 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32916
I20260430 07:57:32.231966 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-1-root/instance:
uuid: "0b70f2147b7a477bb94eedf1daf42b28"
format_stamp: "Formatted at 2026-04-30 07:57:32 on dist-test-slave-6ct4"
I20260430 07:57:32.236934 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260430 07:57:32.241112 9383 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:32.243808 8670 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260430 07:57:32.244462 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-1-root
uuid: "0b70f2147b7a477bb94eedf1daf42b28"
format_stamp: "Formatted at 2026-04-30 07:57:32 on dist-test-slave-6ct4"
I20260430 07:57:32.244771 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:57:32.272059 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:57:32.273367 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:57:32.274978 8670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 07:57:32.277175 8670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 07:57:32.277500 8670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:32.277932 8670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 07:57:32.278131 8670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:32.322211 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.130:35267
I20260430 07:57:32.322258 9445 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.130:35267 every 8 connection(s)
I20260430 07:57:32.326993 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:57:32.335572 9449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:32.336030 9450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:57:32.338253 9452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:57:32.342995 8670 server_base.cc:1061] running on GCE node
I20260430 07:57:32.344354 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:57:32.344648 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:57:32.344877 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535852344854 us; error 0 us; skew 500 ppm
I20260430 07:57:32.345733 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:57:32.345938 9446 heartbeater.cc:344] Connected to a master server at 127.8.119.190:46599
I20260430 07:57:32.346378 9446 heartbeater.cc:461] Registering TS with master...
I20260430 07:57:32.347548 9446 heartbeater.cc:507] Master 127.8.119.190:46599 requested a full tablet report, sending...
I20260430 07:57:32.348659 8670 webserver.cc:492] Webserver started at http://127.8.119.131:34097/ using document root <none> and password file <none>
I20260430 07:57:32.349234 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:57:32.349514 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:57:32.349781 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:57:32.349907 9244 ts_manager.cc:194] Registered new tserver with Master: 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:32.350988 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-2-root/instance:
uuid: "328240de7ae842ac86e6351bf611e935"
format_stamp: "Formatted at 2026-04-30 07:57:32 on dist-test-slave-6ct4"
I20260430 07:57:32.351320 9244 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32930
I20260430 07:57:32.356606 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260430 07:57:32.360647 9457 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:32.361882 8670 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260430 07:57:32.362358 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-2-root
uuid: "328240de7ae842ac86e6351bf611e935"
format_stamp: "Formatted at 2026-04-30 07:57:32 on dist-test-slave-6ct4"
I20260430 07:57:32.362962 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1777535818658779-8670-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:57:32.384126 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:57:32.385536 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:57:32.386976 8670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 07:57:32.389290 8670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 07:57:32.389542 8670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:32.389804 8670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 07:57:32.389981 8670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 07:57:32.441766 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.131:44085
I20260430 07:57:32.441732 9519 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.131:44085 every 8 connection(s)
I20260430 07:57:32.461750 9520 heartbeater.cc:344] Connected to a master server at 127.8.119.190:46599
I20260430 07:57:32.462262 9520 heartbeater.cc:461] Registering TS with master...
I20260430 07:57:32.463722 9520 heartbeater.cc:507] Master 127.8.119.190:46599 requested a full tablet report, sending...
I20260430 07:57:32.466768 9244 ts_manager.cc:194] Registered new tserver with Master: 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:32.467830 8670 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.021215773s
I20260430 07:57:32.468739 9244 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:32936
I20260430 07:57:32.500721 9244 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32946:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 07:57:32.503723 9244 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 07:57:32.579687 9484 tablet_service.cc:1511] Processing CreateTablet for tablet 7c6eda0175e14df28645991383e9f32c (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260430 07:57:32.581718 9484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6eda0175e14df28645991383e9f32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.583027 9336 tablet_service.cc:1511] Processing CreateTablet for tablet b6fa7604ad9b4a1a8e3323e6798f26ac (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260430 07:57:32.584391 9336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fa7604ad9b4a1a8e3323e6798f26ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.584633 9335 tablet_service.cc:1511] Processing CreateTablet for tablet 7c6eda0175e14df28645991383e9f32c (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260430 07:57:32.586513 9334 tablet_service.cc:1511] Processing CreateTablet for tablet 904b5383e0484763a22dffe232eae9e9 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260430 07:57:32.588186 9485 tablet_service.cc:1511] Processing CreateTablet for tablet b6fa7604ad9b4a1a8e3323e6798f26ac (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260430 07:57:32.590587 9408 tablet_service.cc:1511] Processing CreateTablet for tablet 234d7020308741a78ee34213c6cca517 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260430 07:57:32.590740 9410 tablet_service.cc:1511] Processing CreateTablet for tablet 7c6eda0175e14df28645991383e9f32c (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260430 07:57:32.592020 9408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234d7020308741a78ee34213c6cca517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.591615 9483 tablet_service.cc:1511] Processing CreateTablet for tablet 904b5383e0484763a22dffe232eae9e9 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260430 07:57:32.592667 9409 tablet_service.cc:1511] Processing CreateTablet for tablet 904b5383e0484763a22dffe232eae9e9 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260430 07:57:32.593567 9411 tablet_service.cc:1511] Processing CreateTablet for tablet b6fa7604ad9b4a1a8e3323e6798f26ac (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260430 07:57:32.594015 9482 tablet_service.cc:1511] Processing CreateTablet for tablet 234d7020308741a78ee34213c6cca517 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260430 07:57:32.594545 9407 tablet_service.cc:1511] Processing CreateTablet for tablet 7b4a4aefee4a479e8b197c722b7ee02a (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260430 07:57:32.596060 9406 tablet_service.cc:1511] Processing CreateTablet for tablet 667ece5d610e4127b462b1148129eca7 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260430 07:57:32.597745 9481 tablet_service.cc:1511] Processing CreateTablet for tablet 7b4a4aefee4a479e8b197c722b7ee02a (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260430 07:57:32.599834 9480 tablet_service.cc:1511] Processing CreateTablet for tablet 667ece5d610e4127b462b1148129eca7 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260430 07:57:32.601557 9405 tablet_service.cc:1511] Processing CreateTablet for tablet 21e1ae9826db429dbb15987b8f56dde4 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260430 07:57:32.601920 9333 tablet_service.cc:1511] Processing CreateTablet for tablet 234d7020308741a78ee34213c6cca517 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260430 07:57:32.603899 9332 tablet_service.cc:1511] Processing CreateTablet for tablet 7b4a4aefee4a479e8b197c722b7ee02a (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260430 07:57:32.605826 9331 tablet_service.cc:1511] Processing CreateTablet for tablet 667ece5d610e4127b462b1148129eca7 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260430 07:57:32.606550 9404 tablet_service.cc:1511] Processing CreateTablet for tablet 44351546bacf443ab17da51749a8ba47 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260430 07:57:32.607486 9485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fa7604ad9b4a1a8e3323e6798f26ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.610256 9409 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 904b5383e0484763a22dffe232eae9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.613722 9483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 904b5383e0484763a22dffe232eae9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.614436 9482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234d7020308741a78ee34213c6cca517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.617913 9411 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fa7604ad9b4a1a8e3323e6798f26ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.618731 9407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b4a4aefee4a479e8b197c722b7ee02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.620235 9480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 667ece5d610e4127b462b1148129eca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.621888 9334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 904b5383e0484763a22dffe232eae9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.623531 9405 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e1ae9826db429dbb15987b8f56dde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.626088 9404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44351546bacf443ab17da51749a8ba47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.626695 9335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6eda0175e14df28645991383e9f32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.630821 9479 tablet_service.cc:1511] Processing CreateTablet for tablet 21e1ae9826db429dbb15987b8f56dde4 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260430 07:57:32.631862 9478 tablet_service.cc:1511] Processing CreateTablet for tablet 44351546bacf443ab17da51749a8ba47 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260430 07:57:32.632007 9328 tablet_service.cc:1511] Processing CreateTablet for tablet 14d40b1b5ff64e3495088699c5874c42 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260430 07:57:32.630821 9330 tablet_service.cc:1511] Processing CreateTablet for tablet 21e1ae9826db429dbb15987b8f56dde4 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260430 07:57:32.633409 9328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d40b1b5ff64e3495088699c5874c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.633749 9406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 667ece5d610e4127b462b1148129eca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.637704 9330 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e1ae9826db429dbb15987b8f56dde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.634498 9329 tablet_service.cc:1511] Processing CreateTablet for tablet 44351546bacf443ab17da51749a8ba47 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260430 07:57:32.639365 9329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44351546bacf443ab17da51749a8ba47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.641922 9333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234d7020308741a78ee34213c6cca517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.645998 9331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 667ece5d610e4127b462b1148129eca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.649966 9332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b4a4aefee4a479e8b197c722b7ee02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.641949 9410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6eda0175e14df28645991383e9f32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.626698 9481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b4a4aefee4a479e8b197c722b7ee02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.662075 9478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44351546bacf443ab17da51749a8ba47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.665195 9403 tablet_service.cc:1511] Processing CreateTablet for tablet 14d40b1b5ff64e3495088699c5874c42 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260430 07:57:32.666426 9477 tablet_service.cc:1511] Processing CreateTablet for tablet 14d40b1b5ff64e3495088699c5874c42 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260430 07:57:32.666958 9403 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d40b1b5ff64e3495088699c5874c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.667814 9477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d40b1b5ff64e3495088699c5874c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.673511 9479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e1ae9826db429dbb15987b8f56dde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.684780 9327 tablet_service.cc:1511] Processing CreateTablet for tablet fb2a4249b52b437387881cca082c5d17 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260430 07:57:32.686372 9327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2a4249b52b437387881cca082c5d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.713230 9402 tablet_service.cc:1511] Processing CreateTablet for tablet fb2a4249b52b437387881cca082c5d17 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260430 07:57:32.719919 9476 tablet_service.cc:1511] Processing CreateTablet for tablet fb2a4249b52b437387881cca082c5d17 (DEFAULT_TABLE table=test-workload [id=0cda343e4dd649ffa389bb51f7413199]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260430 07:57:32.721588 9476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2a4249b52b437387881cca082c5d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.733989 9402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb2a4249b52b437387881cca082c5d17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:57:32.735896 9535 tablet_bootstrap.cc:492] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28: Bootstrap starting.
I20260430 07:57:32.746976 9536 tablet_bootstrap.cc:492] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad: Bootstrap starting.
I20260430 07:57:32.782968 9535 tablet_bootstrap.cc:654] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.802461 9537 tablet_bootstrap.cc:492] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935: Bootstrap starting.
I20260430 07:57:32.807960 9536 tablet_bootstrap.cc:654] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.811465 9537 tablet_bootstrap.cc:654] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.812928 9535 tablet_bootstrap.cc:492] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28: No bootstrap required, opened a new log
I20260430 07:57:32.814430 9535 ts_tablet_manager.cc:1403] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent bootstrapping tablet: real 0.079s user 0.018s sys 0.000s
I20260430 07:57:32.816459 9536 tablet_bootstrap.cc:492] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad: No bootstrap required, opened a new log
I20260430 07:57:32.817166 9536 ts_tablet_manager.cc:1403] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.009s
I20260430 07:57:32.818712 9537 tablet_bootstrap.cc:492] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935: No bootstrap required, opened a new log
I20260430 07:57:32.818367 9535 raft_consensus.cc:359] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.819188 9537 ts_tablet_manager.cc:1403] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260430 07:57:32.819159 9535 raft_consensus.cc:385] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:32.819650 9535 raft_consensus.cc:740] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Initialized, Role: FOLLOWER
I20260430 07:57:32.819497 9536 raft_consensus.cc:359] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.820320 9536 raft_consensus.cc:385] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:32.820663 9536 raft_consensus.cc:740] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Initialized, Role: FOLLOWER
I20260430 07:57:32.820519 9535 consensus_queue.cc:260] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.821386 9536 consensus_queue.cc:260] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.822474 9537 raft_consensus.cc:359] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.823311 9537 raft_consensus.cc:385] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:32.823722 9535 ts_tablet_manager.cc:1434] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent starting tablet: real 0.009s user 0.000s sys 0.008s
I20260430 07:57:32.823652 9537 raft_consensus.cc:740] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Initialized, Role: FOLLOWER
I20260430 07:57:32.824702 9446 heartbeater.cc:499] Master 127.8.119.190:46599 was elected leader, sending a full tablet report...
I20260430 07:57:32.824795 9537 consensus_queue.cc:260] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
W20260430 07:57:32.827595 9447 tablet.cc:2404] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 07:57:32.827960 9371 heartbeater.cc:499] Master 127.8.119.190:46599 was elected leader, sending a full tablet report...
I20260430 07:57:32.831529 9535 tablet_bootstrap.cc:492] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28: Bootstrap starting.
I20260430 07:57:32.842741 9520 heartbeater.cc:499] Master 127.8.119.190:46599 was elected leader, sending a full tablet report...
I20260430 07:57:32.843200 9537 ts_tablet_manager.cc:1434] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935: Time spent starting tablet: real 0.024s user 0.010s sys 0.014s
I20260430 07:57:32.843744 9535 tablet_bootstrap.cc:654] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.844622 9537 tablet_bootstrap.cc:492] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935: Bootstrap starting.
I20260430 07:57:32.846024 9536 ts_tablet_manager.cc:1434] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent starting tablet: real 0.028s user 0.008s sys 0.002s
I20260430 07:57:32.848425 9536 tablet_bootstrap.cc:492] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad: Bootstrap starting.
I20260430 07:57:32.857707 9536 tablet_bootstrap.cc:654] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.858465 9537 tablet_bootstrap.cc:654] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.863674 9535 tablet_bootstrap.cc:492] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28: No bootstrap required, opened a new log
I20260430 07:57:32.864976 9536 tablet_bootstrap.cc:492] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad: No bootstrap required, opened a new log
I20260430 07:57:32.869519 9535 ts_tablet_manager.cc:1403] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28: Time spent bootstrapping tablet: real 0.038s user 0.020s sys 0.004s
I20260430 07:57:32.869649 9536 ts_tablet_manager.cc:1403] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.006s
I20260430 07:57:32.871199 9537 tablet_bootstrap.cc:492] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935: No bootstrap required, opened a new log
I20260430 07:57:32.871752 9537 ts_tablet_manager.cc:1403] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.008s
I20260430 07:57:32.872182 9536 raft_consensus.cc:359] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:32.872814 9536 raft_consensus.cc:385] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:32.872597 9535 raft_consensus.cc:359] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:32.873113 9536 raft_consensus.cc:740] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Initialized, Role: FOLLOWER
I20260430 07:57:32.873389 9535 raft_consensus.cc:385] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:32.873735 9535 raft_consensus.cc:740] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Initialized, Role: FOLLOWER
I20260430 07:57:32.873867 9536 consensus_queue.cc:260] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:32.874508 9535 consensus_queue.cc:260] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:32.875377 9537 raft_consensus.cc:359] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.876250 9537 raft_consensus.cc:385] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:32.876627 9536 ts_tablet_manager.cc:1434] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260430 07:57:32.876588 9537 raft_consensus.cc:740] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Initialized, Role: FOLLOWER
I20260430 07:57:32.877751 9536 tablet_bootstrap.cc:492] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad: Bootstrap starting.
I20260430 07:57:32.878294 9537 consensus_queue.cc:260] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.877777 9535 ts_tablet_manager.cc:1434] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260430 07:57:32.884126 9535 tablet_bootstrap.cc:492] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28: Bootstrap starting.
I20260430 07:57:32.885572 9536 tablet_bootstrap.cc:654] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.881065 9537 ts_tablet_manager.cc:1434] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260430 07:57:32.887276 9537 tablet_bootstrap.cc:492] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935: Bootstrap starting.
I20260430 07:57:32.893895 9541 raft_consensus.cc:493] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:32.894173 9537 tablet_bootstrap.cc:654] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.894407 9541 raft_consensus.cc:515] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.897117 9541 leader_election.cc:290] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:32.902029 9535 tablet_bootstrap.cc:654] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.915452 9543 raft_consensus.cc:493] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:32.916208 9543 raft_consensus.cc:515] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.922338 9543 leader_election.cc:290] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:32.935784 9536 tablet_bootstrap.cc:492] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad: No bootstrap required, opened a new log
I20260430 07:57:32.936509 9536 ts_tablet_manager.cc:1403] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent bootstrapping tablet: real 0.059s user 0.010s sys 0.009s
I20260430 07:57:32.945109 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "234d7020308741a78ee34213c6cca517" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:32.946981 9536 raft_consensus.cc:359] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.947937 9536 raft_consensus.cc:385] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:32.948239 9536 raft_consensus.cc:740] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Initialized, Role: FOLLOWER
I20260430 07:57:32.949144 9537 tablet_bootstrap.cc:492] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935: No bootstrap required, opened a new log
I20260430 07:57:32.948941 9536 consensus_queue.cc:260] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
W20260430 07:57:32.949235 9386 leader_election.cc:343] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:32.950119 9537 ts_tablet_manager.cc:1403] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.028s
I20260430 07:57:32.953101 9537 raft_consensus.cc:359] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.953874 9535 tablet_bootstrap.cc:492] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28: No bootstrap required, opened a new log
I20260430 07:57:32.954492 9535 ts_tablet_manager.cc:1403] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent bootstrapping tablet: real 0.071s user 0.032s sys 0.015s
I20260430 07:57:32.957430 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "234d7020308741a78ee34213c6cca517" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:32.957607 9535 raft_consensus.cc:359] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.958276 9535 raft_consensus.cc:385] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:32.958626 9535 raft_consensus.cc:740] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Initialized, Role: FOLLOWER
W20260430 07:57:32.959553 9386 leader_election.cc:343] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:32.959352 9535 consensus_queue.cc:260] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.960183 9386 leader_election.cc:304] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28; no voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:32.961119 9536 ts_tablet_manager.cc:1434] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260430 07:57:32.961042 9541 raft_consensus.cc:2749] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:32.953897 9537 raft_consensus.cc:385] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:32.961966 9535 ts_tablet_manager.cc:1434] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260430 07:57:32.961989 9537 raft_consensus.cc:740] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Initialized, Role: FOLLOWER
I20260430 07:57:32.963219 9535 tablet_bootstrap.cc:492] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28: Bootstrap starting.
I20260430 07:57:32.962949 9537 consensus_queue.cc:260] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:32.965206 9537 ts_tablet_manager.cc:1434] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
I20260430 07:57:32.966574 9537 tablet_bootstrap.cc:492] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935: Bootstrap starting.
I20260430 07:57:32.968524 9536 tablet_bootstrap.cc:492] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad: Bootstrap starting.
I20260430 07:57:32.970464 9535 tablet_bootstrap.cc:654] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.974128 9542 raft_consensus.cc:493] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:32.974895 9542 raft_consensus.cc:515] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:32.976943 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "667ece5d610e4127b462b1148129eca7" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
W20260430 07:57:32.978402 9460 leader_election.cc:343] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:32.981110 9537 tablet_bootstrap.cc:654] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.985472 9542 leader_election.cc:290] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:32.996310 9536 tablet_bootstrap.cc:654] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:32.998025 9543 raft_consensus.cc:493] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:32.999732 9543 raft_consensus.cc:515] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.002068 9543 leader_election.cc:290] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.004184 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
W20260430 07:57:33.005648 9460 leader_election.cc:343] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.005616 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "667ece5d610e4127b462b1148129eca7" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.007447 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
W20260430 07:57:33.007627 9461 leader_election.cc:343] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.007920 9537 tablet_bootstrap.cc:492] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935: No bootstrap required, opened a new log
I20260430 07:57:33.008261 9461 leader_election.cc:304] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 328240de7ae842ac86e6351bf611e935; no voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.008508 9537 ts_tablet_manager.cc:1403] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935: Time spent bootstrapping tablet: real 0.042s user 0.019s sys 0.012s
I20260430 07:57:33.009037 9543 raft_consensus.cc:2749] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260430 07:57:33.009536 9461 leader_election.cc:343] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.009943 9461 leader_election.cc:304] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 328240de7ae842ac86e6351bf611e935; no voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.010840 9543 raft_consensus.cc:2749] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:33.011438 9537 raft_consensus.cc:359] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:33.012004 9537 raft_consensus.cc:385] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.012267 9537 raft_consensus.cc:740] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.012882 9537 consensus_queue.cc:260] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:33.012938 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6fa7604ad9b4a1a8e3323e6798f26ac" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.013916 9495 raft_consensus.cc:2468] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 0.
I20260430 07:57:33.015291 9537 ts_tablet_manager.cc:1434] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260430 07:57:33.016623 9311 leader_election.cc:304] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.017690 9542 raft_consensus.cc:2804] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.018061 9542 raft_consensus.cc:493] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.018381 9542 raft_consensus.cc:3060] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:57:33.023715 9292 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:57:33.024075 9535 tablet_bootstrap.cc:492] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28: No bootstrap required, opened a new log
I20260430 07:57:33.024621 9535 ts_tablet_manager.cc:1403] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28: Time spent bootstrapping tablet: real 0.062s user 0.020s sys 0.014s
I20260430 07:57:33.027839 9535 raft_consensus.cc:359] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.028064 9542 raft_consensus.cc:515] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:33.028590 9535 raft_consensus.cc:385] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.028927 9535 raft_consensus.cc:740] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.031306 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6fa7604ad9b4a1a8e3323e6798f26ac" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:33.031946 9495 raft_consensus.cc:3060] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.033881 9537 tablet_bootstrap.cc:492] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935: Bootstrap starting.
I20260430 07:57:33.033494 9535 consensus_queue.cc:260] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.034727 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6fa7604ad9b4a1a8e3323e6798f26ac" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.035360 9420 raft_consensus.cc:2468] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 0.
I20260430 07:57:33.038251 9535 ts_tablet_manager.cc:1434] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20260430 07:57:33.040944 9535 tablet_bootstrap.cc:492] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28: Bootstrap starting.
I20260430 07:57:33.044808 9537 tablet_bootstrap.cc:654] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.047935 9535 tablet_bootstrap.cc:654] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.057317 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6fa7604ad9b4a1a8e3323e6798f26ac" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.057996 9420 raft_consensus.cc:3060] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.058363 9537 tablet_bootstrap.cc:492] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935: No bootstrap required, opened a new log
I20260430 07:57:33.058938 9537 ts_tablet_manager.cc:1403] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935: Time spent bootstrapping tablet: real 0.025s user 0.020s sys 0.003s
I20260430 07:57:33.059644 9535 tablet_bootstrap.cc:492] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28: No bootstrap required, opened a new log
I20260430 07:57:33.060313 9535 ts_tablet_manager.cc:1403] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.012s
I20260430 07:57:33.062870 9537 raft_consensus.cc:359] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.063336 9535 raft_consensus.cc:359] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.064263 9537 raft_consensus.cc:385] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.064451 9535 raft_consensus.cc:385] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.064816 9537 raft_consensus.cc:740] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.064935 9535 raft_consensus.cc:740] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.066008 9535 consensus_queue.cc:260] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.066138 9537 consensus_queue.cc:260] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.067049 9420 raft_consensus.cc:2468] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 1.
I20260430 07:57:33.068979 9537 ts_tablet_manager.cc:1434] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20260430 07:57:33.069832 9495 raft_consensus.cc:2468] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 1.
I20260430 07:57:33.072041 9537 tablet_bootstrap.cc:492] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935: Bootstrap starting.
I20260430 07:57:33.072016 9312 leader_election.cc:304] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.075415 9536 tablet_bootstrap.cc:492] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad: No bootstrap required, opened a new log
I20260430 07:57:33.075992 9536 ts_tablet_manager.cc:1403] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad: Time spent bootstrapping tablet: real 0.108s user 0.026s sys 0.028s
I20260430 07:57:33.070273 9542 leader_election.cc:290] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 election: Requested vote from peers 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.070490 9535 ts_tablet_manager.cc:1434] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260430 07:57:33.078754 9535 tablet_bootstrap.cc:492] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28: Bootstrap starting.
I20260430 07:57:33.078998 9536 raft_consensus.cc:359] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.079821 9536 raft_consensus.cc:385] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.080183 9536 raft_consensus.cc:740] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.080384 9537 tablet_bootstrap.cc:654] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.080991 9536 consensus_queue.cc:260] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.087185 9535 tablet_bootstrap.cc:654] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.087970 9542 raft_consensus.cc:2804] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:33.093513 9542 raft_consensus.cc:697] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:33.095031 9542 consensus_queue.cc:237] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:33.111471 9536 ts_tablet_manager.cc:1434] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad: Time spent starting tablet: real 0.035s user 0.021s sys 0.013s
I20260430 07:57:33.112813 9536 tablet_bootstrap.cc:492] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad: Bootstrap starting.
I20260430 07:57:33.113052 9537 tablet_bootstrap.cc:492] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935: No bootstrap required, opened a new log
I20260430 07:57:33.113665 9537 ts_tablet_manager.cc:1403] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.026s
I20260430 07:57:33.116797 9535 tablet_bootstrap.cc:492] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28: No bootstrap required, opened a new log
I20260430 07:57:33.116852 9537 raft_consensus.cc:359] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.117509 9535 ts_tablet_manager.cc:1403] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.005s
I20260430 07:57:33.117908 9537 raft_consensus.cc:385] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.118391 9537 raft_consensus.cc:740] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.117784 9242 catalog_manager.cc:5671] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 1 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:33.119184 9536 tablet_bootstrap.cc:654] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.119288 9537 consensus_queue.cc:260] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.120601 9535 raft_consensus.cc:359] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.121392 9535 raft_consensus.cc:385] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.121737 9535 raft_consensus.cc:740] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.122599 9535 consensus_queue.cc:260] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.123281 9543 raft_consensus.cc:493] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.123728 9543 raft_consensus.cc:515] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.124698 9537 ts_tablet_manager.cc:1434] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935: Time spent starting tablet: real 0.011s user 0.000s sys 0.006s
I20260430 07:57:33.125241 9535 ts_tablet_manager.cc:1434] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260430 07:57:33.125874 9537 tablet_bootstrap.cc:492] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935: Bootstrap starting.
I20260430 07:57:33.126125 9535 tablet_bootstrap.cc:492] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28: Bootstrap starting.
I20260430 07:57:33.126155 9543 leader_election.cc:290] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.127879 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:33.128291 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.128921 9420 raft_consensus.cc:2468] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 0.
W20260430 07:57:33.129521 9460 leader_election.cc:343] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.131166 9461 leader_election.cc:304] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters: 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.132786 9543 raft_consensus.cc:2804] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.133348 9543 raft_consensus.cc:493] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.133816 9543 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.135943 9535 tablet_bootstrap.cc:654] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.132841 9537 tablet_bootstrap.cc:654] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.143698 9542 raft_consensus.cc:493] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.144515 9542 raft_consensus.cc:515] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.147706 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d40b1b5ff64e3495088699c5874c42" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.147858 9543 raft_consensus.cc:515] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
W20260430 07:57:33.149200 9312 leader_election.cc:343] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.151086 9542 leader_election.cc:290] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.151793 9543 leader_election.cc:290] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.150945 9536 tablet_bootstrap.cc:492] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad: No bootstrap required, opened a new log
I20260430 07:57:33.152587 9536 ts_tablet_manager.cc:1403] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.006s
I20260430 07:57:33.151150 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:33.151328 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.150877 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d40b1b5ff64e3495088699c5874c42" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.159402 9420 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:57:33.160180 9311 leader_election.cc:343] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.160831 9311 leader_election.cc:304] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48cc0d3c7a74492b9f815c91740b38ad; no voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935
I20260430 07:57:33.163230 9542 raft_consensus.cc:2749] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260430 07:57:33.166447 9460 leader_election.cc:343] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.173454 9543 raft_consensus.cc:493] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.174079 9543 raft_consensus.cc:515] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.174436 9420 raft_consensus.cc:2468] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 1.
I20260430 07:57:33.171833 9537 tablet_bootstrap.cc:492] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935: No bootstrap required, opened a new log
I20260430 07:57:33.175617 9537 ts_tablet_manager.cc:1403] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.013s
I20260430 07:57:33.176067 9535 tablet_bootstrap.cc:492] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28: No bootstrap required, opened a new log
I20260430 07:57:33.175966 9461 leader_election.cc:304] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters: 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.176581 9535 ts_tablet_manager.cc:1403] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent bootstrapping tablet: real 0.051s user 0.017s sys 0.018s
I20260430 07:57:33.179522 9535 raft_consensus.cc:359] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.178661 9537 raft_consensus.cc:359] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.180461 9535 raft_consensus.cc:385] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.180797 9537 raft_consensus.cc:385] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.180876 9535 raft_consensus.cc:740] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.181318 9537 raft_consensus.cc:740] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.181994 9535 consensus_queue.cc:260] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.183218 9543 leader_election.cc:290] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.182134 9537 consensus_queue.cc:260] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.183499 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "234d7020308741a78ee34213c6cca517" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:33.169008 9536 raft_consensus.cc:359] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.184645 9536 raft_consensus.cc:385] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.184935 9536 raft_consensus.cc:740] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.185230 9537 ts_tablet_manager.cc:1434] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
W20260430 07:57:33.185907 9460 leader_election.cc:343] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.185794 9536 consensus_queue.cc:260] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.186803 9535 ts_tablet_manager.cc:1434] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent starting tablet: real 0.010s user 0.002s sys 0.005s
I20260430 07:57:33.187803 9535 tablet_bootstrap.cc:492] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28: Bootstrap starting.
I20260430 07:57:33.188357 9536 ts_tablet_manager.cc:1434] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20260430 07:57:33.189405 9536 tablet_bootstrap.cc:492] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad: Bootstrap starting.
I20260430 07:57:33.183781 9543 raft_consensus.cc:2804] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:33.194308 9535 tablet_bootstrap.cc:654] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.195817 9536 tablet_bootstrap.cc:654] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.200213 9537 tablet_bootstrap.cc:492] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935: Bootstrap starting.
I20260430 07:57:33.207839 9537 tablet_bootstrap.cc:654] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.212881 9536 tablet_bootstrap.cc:492] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad: No bootstrap required, opened a new log
I20260430 07:57:33.213727 9536 ts_tablet_manager.cc:1403] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.006s
I20260430 07:57:33.216634 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "234d7020308741a78ee34213c6cca517" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.217602 9420 raft_consensus.cc:2468] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 0.
I20260430 07:57:33.219620 9461 leader_election.cc:304] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters: 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.220124 9536 raft_consensus.cc:359] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.220745 9535 tablet_bootstrap.cc:492] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28: No bootstrap required, opened a new log
I20260430 07:57:33.220961 9573 raft_consensus.cc:2804] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.221312 9535 ts_tablet_manager.cc:1403] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.008s
I20260430 07:57:33.221484 9573 raft_consensus.cc:493] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.221889 9573 raft_consensus.cc:3060] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.221088 9536 raft_consensus.cc:385] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.222559 9536 raft_consensus.cc:740] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.223466 9536 consensus_queue.cc:260] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.224287 9537 tablet_bootstrap.cc:492] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935: No bootstrap required, opened a new log
I20260430 07:57:33.224090 9535 raft_consensus.cc:359] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.224766 9535 raft_consensus.cc:385] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.225057 9535 raft_consensus.cc:740] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.226579 9543 raft_consensus.cc:697] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Becoming Leader. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:33.226958 9535 consensus_queue.cc:260] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.227505 9543 consensus_queue.cc:237] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.230039 9573 raft_consensus.cc:515] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.226363 9536 ts_tablet_manager.cc:1434] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260430 07:57:33.232106 9536 tablet_bootstrap.cc:492] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad: Bootstrap starting.
I20260430 07:57:33.232769 9535 ts_tablet_manager.cc:1434] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260430 07:57:33.233165 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "234d7020308741a78ee34213c6cca517" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:33.233741 9535 tablet_bootstrap.cc:492] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28: Bootstrap starting.
W20260430 07:57:33.234546 9460 leader_election.cc:343] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.236686 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "234d7020308741a78ee34213c6cca517" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.237324 9420 raft_consensus.cc:3060] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.241663 9536 tablet_bootstrap.cc:654] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.243656 9535 tablet_bootstrap.cc:654] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.245167 9420 raft_consensus.cc:2468] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 1.
I20260430 07:57:33.246623 9461 leader_election.cc:304] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters: 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.247675 9574 raft_consensus.cc:2804] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:33.249169 9537 ts_tablet_manager.cc:1403] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.000s
I20260430 07:57:33.249419 9573 leader_election.cc:290] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.250972 9574 raft_consensus.cc:697] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Becoming Leader. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:33.251849 9537 raft_consensus.cc:359] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.251856 9574 consensus_queue.cc:237] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.252607 9537 raft_consensus.cc:385] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.252911 9537 raft_consensus.cc:740] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.253662 9537 consensus_queue.cc:260] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.255795 9537 ts_tablet_manager.cc:1434] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260430 07:57:33.256742 9537 tablet_bootstrap.cc:492] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935: Bootstrap starting.
I20260430 07:57:33.261420 9535 tablet_bootstrap.cc:492] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28: No bootstrap required, opened a new log
I20260430 07:57:33.261905 9535 ts_tablet_manager.cc:1403] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent bootstrapping tablet: real 0.028s user 0.013s sys 0.003s
I20260430 07:57:33.262976 9537 tablet_bootstrap.cc:654] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.264549 9535 raft_consensus.cc:359] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.269537 9537 tablet_bootstrap.cc:492] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935: No bootstrap required, opened a new log
I20260430 07:57:33.265239 9535 raft_consensus.cc:385] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.265663 9244 catalog_manager.cc:5671] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 reported cstate change: term changed from 0 to 1, leader changed from <none> to 328240de7ae842ac86e6351bf611e935 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "328240de7ae842ac86e6351bf611e935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:33.269975 9537 ts_tablet_manager.cc:1403] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260430 07:57:33.270118 9535 raft_consensus.cc:740] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.271322 9536 tablet_bootstrap.cc:492] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad: No bootstrap required, opened a new log
I20260430 07:57:33.270893 9535 consensus_queue.cc:260] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.271795 9536 ts_tablet_manager.cc:1403] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad: Time spent bootstrapping tablet: real 0.040s user 0.025s sys 0.007s
I20260430 07:57:33.272858 9537 raft_consensus.cc:359] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.273524 9537 raft_consensus.cc:385] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.273787 9537 raft_consensus.cc:740] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.274731 9536 raft_consensus.cc:359] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.275404 9536 raft_consensus.cc:385] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.275692 9536 raft_consensus.cc:740] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.276510 9535 ts_tablet_manager.cc:1434] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s
I20260430 07:57:33.276355 9536 consensus_queue.cc:260] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.277616 9535 tablet_bootstrap.cc:492] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28: Bootstrap starting.
I20260430 07:57:33.278139 9574 raft_consensus.cc:493] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.278863 9536 ts_tablet_manager.cc:1434] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260430 07:57:33.278630 9574 raft_consensus.cc:515] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.280107 9536 tablet_bootstrap.cc:492] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad: Bootstrap starting.
I20260430 07:57:33.280786 9574 leader_election.cc:290] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.282521 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:33.283285 9346 raft_consensus.cc:2468] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 0.
I20260430 07:57:33.283644 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.284333 9420 raft_consensus.cc:2468] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 0.
I20260430 07:57:33.284657 9460 leader_election.cc:304] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.285634 9574 raft_consensus.cc:2804] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.286077 9574 raft_consensus.cc:493] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.286477 9574 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.287442 9536 tablet_bootstrap.cc:654] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.274617 9537 consensus_queue.cc:260] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.294534 9574 raft_consensus.cc:515] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.302606 9537 ts_tablet_manager.cc:1434] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260430 07:57:33.304128 9537 tablet_bootstrap.cc:492] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935: Bootstrap starting.
I20260430 07:57:33.305955 9536 tablet_bootstrap.cc:492] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad: No bootstrap required, opened a new log
I20260430 07:57:33.305295 9241 catalog_manager.cc:5671] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 reported cstate change: term changed from 0 to 1, leader changed from <none> to 328240de7ae842ac86e6351bf611e935 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "328240de7ae842ac86e6351bf611e935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:33.306479 9536 ts_tablet_manager.cc:1403] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.008s
I20260430 07:57:33.309345 9536 raft_consensus.cc:359] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.310187 9536 raft_consensus.cc:385] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.310451 9536 raft_consensus.cc:740] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.310487 9541 raft_consensus.cc:493] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.311095 9541 raft_consensus.cc:515] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.311228 9536 consensus_queue.cc:260] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.311837 9574 leader_election.cc:290] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.303033 9535 tablet_bootstrap.cc:654] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.313832 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.314517 9420 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.314893 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:33.315478 9346 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.317437 9536 ts_tablet_manager.cc:1434] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260430 07:57:33.318665 9536 tablet_bootstrap.cc:492] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad: Bootstrap starting.
I20260430 07:57:33.322253 9574 raft_consensus.cc:493] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.322944 9574 raft_consensus.cc:515] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.326184 9536 tablet_bootstrap.cc:654] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.327203 9345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d40b1b5ff64e3495088699c5874c42" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:33.329171 9345 raft_consensus.cc:2468] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 0.
I20260430 07:57:33.331614 9541 leader_election.cc:290] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.334548 9542 raft_consensus.cc:493] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.335369 9542 raft_consensus.cc:515] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.335769 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.334733 9574 leader_election.cc:290] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.337159 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d40b1b5ff64e3495088699c5874c42" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
W20260430 07:57:33.337438 9386 leader_election.cc:343] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.337709 9542 leader_election.cc:290] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.337932 9421 raft_consensus.cc:2468] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 0.
I20260430 07:57:33.339293 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.339897 9421 raft_consensus.cc:2468] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 0.
I20260430 07:57:33.341194 9312 leader_election.cc:304] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.342058 9542 raft_consensus.cc:2804] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.342447 9542 raft_consensus.cc:493] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.342800 9542 raft_consensus.cc:3060] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.343147 9537 tablet_bootstrap.cc:654] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.346323 9345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:33.337474 9460 leader_election.cc:304] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.350488 9574 raft_consensus.cc:2804] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.351015 9574 raft_consensus.cc:493] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.351473 9574 raft_consensus.cc:3060] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.355353 9420 raft_consensus.cc:2468] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 1.
I20260430 07:57:33.343842 9346 raft_consensus.cc:2468] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 1.
I20260430 07:57:33.357880 9461 leader_election.cc:304] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters:
I20260430 07:57:33.356557 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.360230 9543 raft_consensus.cc:2804] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:33.360880 9543 raft_consensus.cc:697] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Becoming Leader. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
W20260430 07:57:33.362871 9311 leader_election.cc:343] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.363161 9535 tablet_bootstrap.cc:492] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28: No bootstrap required, opened a new log
I20260430 07:57:33.363816 9535 ts_tablet_manager.cc:1403] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent bootstrapping tablet: real 0.086s user 0.014s sys 0.017s
I20260430 07:57:33.363655 9543 consensus_queue.cc:237] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.366472 9563 raft_consensus.cc:493] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.367116 9563 raft_consensus.cc:515] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.369678 9563 leader_election.cc:290] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.370771 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "234d7020308741a78ee34213c6cca517" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.371913 9420 raft_consensus.cc:2393] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48cc0d3c7a74492b9f815c91740b38ad in current term 1: Already voted for candidate 328240de7ae842ac86e6351bf611e935 in this term.
I20260430 07:57:33.372323 9536 tablet_bootstrap.cc:492] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad: No bootstrap required, opened a new log
I20260430 07:57:33.372839 9536 ts_tablet_manager.cc:1403] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent bootstrapping tablet: real 0.054s user 0.019s sys 0.008s
I20260430 07:57:33.373152 9535 raft_consensus.cc:359] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.374099 9535 raft_consensus.cc:385] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.374418 9535 raft_consensus.cc:740] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.374593 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "234d7020308741a78ee34213c6cca517" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.375319 9535 consensus_queue.cc:260] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.376310 9311 leader_election.cc:304] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48cc0d3c7a74492b9f815c91740b38ad; no voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935
I20260430 07:57:33.377436 9563 raft_consensus.cc:3060] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.381213 9536 raft_consensus.cc:359] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.382136 9536 raft_consensus.cc:385] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.381006 9240 catalog_manager.cc:5671] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 reported cstate change: term changed from 0 to 1, leader changed from <none> to 328240de7ae842ac86e6351bf611e935 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "328240de7ae842ac86e6351bf611e935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:33.382445 9536 raft_consensus.cc:740] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.383313 9574 raft_consensus.cc:515] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.383471 9536 consensus_queue.cc:260] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.386246 9536 ts_tablet_manager.cc:1434] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent starting tablet: real 0.013s user 0.004s sys 0.004s
I20260430 07:57:33.387238 9536 tablet_bootstrap.cc:492] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad: Bootstrap starting.
I20260430 07:57:33.388165 9535 ts_tablet_manager.cc:1434] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28: Time spent starting tablet: real 0.024s user 0.009s sys 0.001s
I20260430 07:57:33.391392 9562 raft_consensus.cc:493] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.391891 9562 raft_consensus.cc:515] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.393517 9542 raft_consensus.cc:515] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.394054 9562 leader_election.cc:290] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.395941 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.397972 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.398218 9536 tablet_bootstrap.cc:654] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad: Neither blocks nor log segments found. Creating new log.
I20260430 07:57:33.398248 9345 raft_consensus.cc:2393] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b70f2147b7a477bb94eedf1daf42b28 in current term 1: Already voted for candidate 48cc0d3c7a74492b9f815c91740b38ad in this term.
I20260430 07:57:33.398759 9420 raft_consensus.cc:2393] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48cc0d3c7a74492b9f815c91740b38ad in current term 1: Already voted for candidate 328240de7ae842ac86e6351bf611e935 in this term.
I20260430 07:57:33.400408 9312 leader_election.cc:304] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48cc0d3c7a74492b9f815c91740b38ad; no voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935
I20260430 07:57:33.401875 9562 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.393105 9563 raft_consensus.cc:2749] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:33.403505 9386 leader_election.cc:304] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28; no voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.404603 9541 raft_consensus.cc:3060] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.407675 9345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d40b1b5ff64e3495088699c5874c42" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:33.408515 9345 raft_consensus.cc:3060] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.416280 9541 raft_consensus.cc:2749] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:33.418720 9543 raft_consensus.cc:493] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.420354 9574 leader_election.cc:290] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.419674 9543 raft_consensus.cc:515] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.421645 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d40b1b5ff64e3495088699c5874c42" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.422086 9542 leader_election.cc:290] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.422259 9543 leader_election.cc:290] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.418125 9541 raft_consensus.cc:493] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.422266 9420 raft_consensus.cc:3060] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.423027 9541 raft_consensus.cc:515] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.423561 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.424140 9562 raft_consensus.cc:2749] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:33.424314 9421 raft_consensus.cc:2468] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 0.
I20260430 07:57:33.425678 9541 leader_election.cc:290] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.426280 9541 raft_consensus.cc:493] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.426304 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:33.426759 9344 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:33.427006 9541 raft_consensus.cc:515] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.427501 9344 raft_consensus.cc:2468] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 0.
W20260430 07:57:33.427690 9311 leader_election.cc:343] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.427768 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.426234 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:33.432479 9461 leader_election.cc:304] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters:
I20260430 07:57:33.433720 9541 leader_election.cc:290] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
W20260430 07:57:33.434866 9386 leader_election.cc:343] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.435393 9537 tablet_bootstrap.cc:492] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935: No bootstrap required, opened a new log
I20260430 07:57:33.435163 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:33.435832 9537 ts_tablet_manager.cc:1403] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935: Time spent bootstrapping tablet: real 0.132s user 0.024s sys 0.023s
I20260430 07:57:33.435947 9346 raft_consensus.cc:2468] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 0.
I20260430 07:57:33.431490 9562 raft_consensus.cc:493] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.428463 9494 raft_consensus.cc:2468] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 0.
I20260430 07:57:33.436904 9562 raft_consensus.cc:515] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.437606 9386 leader_election.cc:304] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.438340 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.438618 9589 raft_consensus.cc:493] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.438529 9386 leader_election.cc:304] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters: 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.439110 9589 raft_consensus.cc:515] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.442102 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.442765 9494 raft_consensus.cc:2468] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 0.
I20260430 07:57:33.443677 9589 leader_election.cc:290] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.443890 9311 leader_election.cc:304] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.444559 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.440207 9543 raft_consensus.cc:2804] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.444962 9542 raft_consensus.cc:2804] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.439978 9562 leader_election.cc:290] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.446245 9542 raft_consensus.cc:493] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.446755 9495 raft_consensus.cc:2468] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 0.
I20260430 07:57:33.447779 9542 raft_consensus.cc:3060] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.448314 9419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.448921 9543 raft_consensus.cc:493] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.449615 9543 raft_consensus.cc:3060] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.449064 9419 raft_consensus.cc:2468] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 0.
I20260430 07:57:33.440812 9541 raft_consensus.cc:2804] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.451719 9541 raft_consensus.cc:493] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.452179 9541 raft_consensus.cc:3060] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.446759 9589 raft_consensus.cc:2804] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.455163 9537 raft_consensus.cc:359] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.455937 9537 raft_consensus.cc:385] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.456218 9537 raft_consensus.cc:740] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.453410 9589 raft_consensus.cc:493] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.457890 9589 raft_consensus.cc:3060] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.457649 9537 consensus_queue.cc:260] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.458408 9345 raft_consensus.cc:2468] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 1.
I20260430 07:57:33.458731 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:33.461875 9537 ts_tablet_manager.cc:1434] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935: Time spent starting tablet: real 0.026s user 0.004s sys 0.003s
I20260430 07:57:33.462975 9421 raft_consensus.cc:2468] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 1.
I20260430 07:57:33.463935 9312 leader_election.cc:304] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters: 328240de7ae842ac86e6351bf611e935
I20260430 07:57:33.464998 9562 raft_consensus.cc:2804] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:33.465502 9562 raft_consensus.cc:697] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:33.466301 9562 consensus_queue.cc:237] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.471508 9460 leader_election.cc:304] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.471752 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.472445 9574 raft_consensus.cc:2804] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:33.472937 9574 raft_consensus.cc:697] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Becoming Leader. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:33.472744 9495 raft_consensus.cc:2468] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 0.
I20260430 07:57:33.473905 9574 consensus_queue.cc:237] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.475796 9386 leader_election.cc:304] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters:
I20260430 07:57:33.479080 9590 raft_consensus.cc:2804] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.479660 9590 raft_consensus.cc:493] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.480080 9590 raft_consensus.cc:3060] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.480368 9589 raft_consensus.cc:515] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.482388 9589 leader_election.cc:290] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.484428 9420 raft_consensus.cc:2468] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 1.
I20260430 07:57:33.487169 9563 consensus_queue.cc:1048] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.489528 9345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:33.489916 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:33.490576 9495 raft_consensus.cc:3060] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:57:33.490903 9386 leader_election.cc:343] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:57:33.497354 9543 raft_consensus.cc:515] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.501262 9536 tablet_bootstrap.cc:492] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad: No bootstrap required, opened a new log
I20260430 07:57:33.501801 9536 ts_tablet_manager.cc:1403] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent bootstrapping tablet: real 0.115s user 0.022s sys 0.001s
I20260430 07:57:33.500313 9240 catalog_manager.cc:5671] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 reported cstate change: term changed from 0 to 1, leader changed from <none> to 328240de7ae842ac86e6351bf611e935 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "328240de7ae842ac86e6351bf611e935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:33.505856 9541 raft_consensus.cc:515] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.507768 9541 leader_election.cc:290] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.508615 9495 raft_consensus.cc:2468] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 1.
I20260430 07:57:33.509976 9536 raft_consensus.cc:359] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.510694 9536 raft_consensus.cc:385] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:57:33.511000 9536 raft_consensus.cc:740] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Initialized, Role: FOLLOWER
I20260430 07:57:33.511883 9536 consensus_queue.cc:260] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.490569 9590 raft_consensus.cc:515] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.500219 9542 raft_consensus.cc:515] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.515146 9345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:33.518028 9536 ts_tablet_manager.cc:1434] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260430 07:57:33.514195 9543 leader_election.cc:290] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.523651 9563 raft_consensus.cc:493] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.523784 9344 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:33.524298 9563 raft_consensus.cc:515] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.517659 9345 raft_consensus.cc:3060] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.515857 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.532280 9563 leader_election.cc:290] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.530138 9346 raft_consensus.cc:2393] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b70f2147b7a477bb94eedf1daf42b28 in current term 1: Already voted for candidate 48cc0d3c7a74492b9f815c91740b38ad in this term.
I20260430 07:57:33.529361 9590 leader_election.cc:290] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.537132 9421 raft_consensus.cc:2393] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 328240de7ae842ac86e6351bf611e935 in current term 1: Already voted for candidate 0b70f2147b7a477bb94eedf1daf42b28 in this term.
I20260430 07:57:33.536481 9386 leader_election.cc:304] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters: 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.540369 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "667ece5d610e4127b462b1148129eca7" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.541245 9344 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:33.541481 9420 raft_consensus.cc:2468] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 0.
I20260430 07:57:33.543203 9312 leader_election.cc:304] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.543707 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:33.544335 9494 raft_consensus.cc:3060] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.532794 9563 consensus_queue.cc:1048] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260430 07:57:33.548866 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:33.552304 9345 raft_consensus.cc:2468] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 1.
I20260430 07:57:33.553454 9495 raft_consensus.cc:2393] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b70f2147b7a477bb94eedf1daf42b28 in current term 1: Already voted for candidate 328240de7ae842ac86e6351bf611e935 in this term.
I20260430 07:57:33.555823 9460 leader_election.cc:304] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters: 0b70f2147b7a477bb94eedf1daf42b28
I20260430 07:57:33.556129 9542 leader_election.cc:290] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.541219 9590 raft_consensus.cc:2804] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:33.556692 9543 raft_consensus.cc:2804] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:33.556665 9542 raft_consensus.cc:2804] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:57:33.559362 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.561080 9543 raft_consensus.cc:697] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Becoming Leader. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:33.561668 9421 raft_consensus.cc:2393] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 48cc0d3c7a74492b9f815c91740b38ad in current term 1: Already voted for candidate 0b70f2147b7a477bb94eedf1daf42b28 in this term.
I20260430 07:57:33.562011 9543 consensus_queue.cc:237] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.563445 9542 raft_consensus.cc:493] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:57:33.563827 9542 raft_consensus.cc:3060] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.565655 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "667ece5d610e4127b462b1148129eca7" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.566366 9495 raft_consensus.cc:2468] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 0.
I20260430 07:57:33.553540 9494 raft_consensus.cc:2468] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 1.
I20260430 07:57:33.572781 9542 raft_consensus.cc:515] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.564127 9241 catalog_manager.cc:5671] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 1 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:33.578846 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "667ece5d610e4127b462b1148129eca7" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.579725 9421 raft_consensus.cc:3060] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.570770 9590 raft_consensus.cc:697] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 LEADER]: Becoming Leader. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Running, Role: LEADER
I20260430 07:57:33.597227 9542 leader_election.cc:290] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.599674 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:33.600839 9494 raft_consensus.cc:2393] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 48cc0d3c7a74492b9f815c91740b38ad in current term 1: Already voted for candidate 0b70f2147b7a477bb94eedf1daf42b28 in this term.
I20260430 07:57:33.607020 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "667ece5d610e4127b462b1148129eca7" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:33.607260 9311 leader_election.cc:304] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48cc0d3c7a74492b9f815c91740b38ad; no voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935
I20260430 07:57:33.607704 9494 raft_consensus.cc:3060] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.608369 9542 raft_consensus.cc:2749] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260430 07:57:33.609295 9590 consensus_queue.cc:237] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.558723 9386 leader_election.cc:304] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28; no voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad
W20260430 07:57:33.612744 9386 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:44085, user_credentials={real_user=slave}} blocked reactor thread for 54146.7us
I20260430 07:57:33.613209 9589 raft_consensus.cc:2749] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260430 07:57:33.613785 9386 leader_election.cc:304] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters: 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.623127 9543 consensus_queue.cc:1048] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.625967 9589 raft_consensus.cc:2804] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:33.627012 9589 raft_consensus.cc:697] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 LEADER]: Becoming Leader. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Running, Role: LEADER
I20260430 07:57:33.627895 9589 consensus_queue.cc:237] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.632259 9494 raft_consensus.cc:2468] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 1.
I20260430 07:57:33.631515 9241 catalog_manager.cc:5671] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 reported cstate change: term changed from 0 to 1, leader changed from <none> to 328240de7ae842ac86e6351bf611e935 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "328240de7ae842ac86e6351bf611e935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:33.620788 9421 raft_consensus.cc:2468] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 1.
I20260430 07:57:33.636917 9312 leader_election.cc:304] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.638793 9597 raft_consensus.cc:2804] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:57:33.639487 9597 raft_consensus.cc:697] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:33.643725 9597 consensus_queue.cc:237] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.666252 9543 consensus_queue.cc:1048] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:33.687677 9244 catalog_manager.cc:5671] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "0b70f2147b7a477bb94eedf1daf42b28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:33.688799 9240 catalog_manager.cc:5671] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 1 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:33.703163 9240 catalog_manager.cc:5671] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "0b70f2147b7a477bb94eedf1daf42b28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:33.732146 9543 consensus_queue.cc:1048] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.746619 9581 consensus_queue.cc:1048] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:33.764683 9494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
mode: GRACEFUL
new_leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:34246
I20260430 07:57:33.765405 9494 raft_consensus.cc:606] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Received request to transfer leadership to TS 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.771930 9494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
mode: GRACEFUL
new_leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:34246
I20260430 07:57:33.772557 9494 raft_consensus.cc:606] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Received request to transfer leadership to TS 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.778726 9494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "234d7020308741a78ee34213c6cca517"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
mode: GRACEFUL
new_leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:34246
I20260430 07:57:33.779341 9494 raft_consensus.cc:606] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Received request to transfer leadership to TS 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.785250 9494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
mode: GRACEFUL
new_leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:34246
I20260430 07:57:33.785907 9494 raft_consensus.cc:606] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Received request to transfer leadership to TS 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.792860 9581 raft_consensus.cc:493] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.793450 9581 raft_consensus.cc:515] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.795568 9581 leader_election.cc:290] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:33.796435 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad" is_pre_election: true
I20260430 07:57:33.796605 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.797753 9421 raft_consensus.cc:2393] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 328240de7ae842ac86e6351bf611e935 in current term 1: Already voted for candidate 48cc0d3c7a74492b9f815c91740b38ad in this term.
I20260430 07:57:33.798738 9461 leader_election.cc:304] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 328240de7ae842ac86e6351bf611e935; no voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:33.799772 9581 raft_consensus.cc:3060] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.801474 9346 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "667ece5d610e4127b462b1148129eca7"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
mode: GRACEFUL
new_leader_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
from {username='slave'} at 127.0.0.1:44552
I20260430 07:57:33.802030 9346 raft_consensus.cc:606] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Received request to transfer leadership to TS 0b70f2147b7a477bb94eedf1daf42b28
I20260430 07:57:33.806209 9581 raft_consensus.cc:2749] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:33.813983 9494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "14d40b1b5ff64e3495088699c5874c42"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
mode: GRACEFUL
new_leader_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
from {username='slave'} at 127.0.0.1:34246
I20260430 07:57:33.814560 9494 raft_consensus.cc:606] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Received request to transfer leadership to TS 0b70f2147b7a477bb94eedf1daf42b28
I20260430 07:57:33.820577 9346 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
mode: GRACEFUL
new_leader_uuid: "328240de7ae842ac86e6351bf611e935"
from {username='slave'} at 127.0.0.1:44552
I20260430 07:57:33.821252 9346 raft_consensus.cc:606] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Received request to transfer leadership to TS 328240de7ae842ac86e6351bf611e935
I20260430 07:57:33.828495 9581 consensus_queue.cc:1048] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.840281 9573 consensus_queue.cc:1048] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.849988 9574 raft_consensus.cc:993] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935: : Instructing follower 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:33.850397 9573 raft_consensus.cc:1081] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Signalling peer 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:33.851977 9346 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:44542
I20260430 07:57:33.852638 9346 raft_consensus.cc:493] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:33.853029 9346 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:33.858526 9346 raft_consensus.cc:515] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.860383 9346 leader_election.cc:290] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.861186 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.861464 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:33.861845 9421 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:33.861961 9494 raft_consensus.cc:3055] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:33.862285 9494 raft_consensus.cc:740] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:33.863090 9494 consensus_queue.cc:260] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.864356 9494 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:33.869226 9421 raft_consensus.cc:2468] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:33.871080 9312 leader_election.cc:304] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.871619 9494 raft_consensus.cc:2468] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:33.871870 9597 raft_consensus.cc:2804] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:33.872351 9597 raft_consensus.cc:697] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:33.873163 9597 consensus_queue.cc:237] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.880431 9240 catalog_manager.cc:5671] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 1 to 2, leader changed from 328240de7ae842ac86e6351bf611e935 (127.8.119.131) to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 2 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:33.883656 9573 consensus_queue.cc:1048] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.894695 9574 consensus_queue.cc:1048] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.903810 9574 raft_consensus.cc:993] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935: : Instructing follower 0b70f2147b7a477bb94eedf1daf42b28 to start an election
I20260430 07:57:33.904275 9574 raft_consensus.cc:1081] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Signalling peer 0b70f2147b7a477bb94eedf1daf42b28 to start an election
I20260430 07:57:33.906018 9421 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "14d40b1b5ff64e3495088699c5874c42"
dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
from {username='slave'} at 127.0.0.1:40602
I20260430 07:57:33.906643 9421 raft_consensus.cc:493] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:33.907033 9421 raft_consensus.cc:3060] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:33.913578 9421 raft_consensus.cc:515] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.915493 9421 leader_election.cc:290] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 2 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.916555 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d40b1b5ff64e3495088699c5874c42" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:33.917238 9346 raft_consensus.cc:3060] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:33.917161 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14d40b1b5ff64e3495088699c5874c42" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:33.917953 9494 raft_consensus.cc:3055] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:33.918280 9494 raft_consensus.cc:740] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:33.919096 9494 consensus_queue.cc:260] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.920478 9494 raft_consensus.cc:3060] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:33.926024 9346 raft_consensus.cc:2468] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 2.
I20260430 07:57:33.927385 9386 leader_election.cc:304] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:33.928177 9590 raft_consensus.cc:2804] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:33.928601 9590 raft_consensus.cc:697] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 LEADER]: Becoming Leader. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Running, Role: LEADER
I20260430 07:57:33.928979 9494 raft_consensus.cc:2468] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 2.
I20260430 07:57:33.929653 9590 consensus_queue.cc:237] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.938076 9597 raft_consensus.cc:493] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:57:33.937636 9240 catalog_manager.cc:5671] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 reported cstate change: term changed from 1 to 2, leader changed from 328240de7ae842ac86e6351bf611e935 (127.8.119.131) to 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "0b70f2147b7a477bb94eedf1daf42b28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:33.938553 9597 raft_consensus.cc:515] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.940270 9597 leader_election.cc:290] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:33.941118 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28" is_pre_election: true
I20260430 07:57:33.941325 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "328240de7ae842ac86e6351bf611e935" is_pre_election: true
I20260430 07:57:33.942000 9494 raft_consensus.cc:2393] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48cc0d3c7a74492b9f815c91740b38ad in current term 1: Already voted for candidate 0b70f2147b7a477bb94eedf1daf42b28 in this term.
I20260430 07:57:33.943384 9311 leader_election.cc:304] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48cc0d3c7a74492b9f815c91740b38ad; no voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935
I20260430 07:57:33.944339 9597 raft_consensus.cc:3060] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:57:33.951081 9597 raft_consensus.cc:2749] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:57:33.962596 9574 consensus_queue.cc:1048] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.967175 9597 consensus_queue.cc:1048] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.968969 9574 consensus_queue.cc:1048] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.974323 9573 raft_consensus.cc:993] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935: : Instructing follower 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:33.975131 9573 raft_consensus.cc:1081] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Signalling peer 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:33.978168 9346 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:44542
I20260430 07:57:33.978824 9346 raft_consensus.cc:493] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:33.979238 9346 raft_consensus.cc:3060] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:33.989671 9346 raft_consensus.cc:515] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.990427 9596 consensus_queue.cc:1048] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:33.992257 9345 raft_consensus.cc:1240] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Rejecting Update request from peer 328240de7ae842ac86e6351bf611e935 for earlier term 1. Current term is 2. Ops: []
I20260430 07:57:33.994771 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:33.995114 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "904b5383e0484763a22dffe232eae9e9" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:33.995374 9420 raft_consensus.cc:3060] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:33.995666 9495 raft_consensus.cc:3055] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:33.996184 9495 raft_consensus.cc:740] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:33.997524 9495 consensus_queue.cc:260] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:33.992316 9346 leader_election.cc:290] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:34.000682 9420 raft_consensus.cc:2468] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:34.002076 9312 leader_election.cc:304] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:34.001978 9574 consensus_queue.cc:1059] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:34.003006 9597 raft_consensus.cc:2804] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:34.004300 9597 raft_consensus.cc:697] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:34.005257 9597 consensus_queue.cc:237] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.006994 9420 raft_consensus.cc:1240] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Rejecting Update request from peer 328240de7ae842ac86e6351bf611e935 for earlier term 1. Current term is 2. Ops: []
I20260430 07:57:34.008909 9596 raft_consensus.cc:993] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad: : Instructing follower 328240de7ae842ac86e6351bf611e935 to start an election
I20260430 07:57:34.009416 9596 raft_consensus.cc:1081] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Signalling peer 328240de7ae842ac86e6351bf611e935 to start an election
I20260430 07:57:34.011246 9494 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
from {username='slave'} at 127.0.0.1:34238
I20260430 07:57:34.011873 9494 raft_consensus.cc:493] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:34.012267 9494 raft_consensus.cc:3060] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.008781 9495 raft_consensus.cc:3060] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.016585 9241 catalog_manager.cc:5671] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 1 to 2, leader changed from 328240de7ae842ac86e6351bf611e935 (127.8.119.131) to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 2 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:34.020742 9494 raft_consensus.cc:515] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.022346 9495 raft_consensus.cc:2468] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:34.023957 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:34.024142 9494 leader_election.cc:290] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 2 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:34.024539 9346 raft_consensus.cc:3055] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:34.024875 9346 raft_consensus.cc:740] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:34.025672 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb2a4249b52b437387881cca082c5d17" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:34.026263 9420 raft_consensus.cc:3060] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.025682 9346 consensus_queue.cc:260] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.028187 9346 raft_consensus.cc:3060] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.032243 9420 raft_consensus.cc:2468] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 2.
I20260430 07:57:34.033416 9461 leader_election.cc:304] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters:
I20260430 07:57:34.034147 9573 raft_consensus.cc:2804] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:34.034626 9573 raft_consensus.cc:697] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 2 LEADER]: Becoming Leader. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:34.035727 9346 raft_consensus.cc:2468] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 2.
I20260430 07:57:34.035437 9573 consensus_queue.cc:237] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.043843 9595 consensus_queue.cc:1048] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:34.046237 9241 catalog_manager.cc:5671] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 reported cstate change: term changed from 1 to 2, leader changed from 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129) to 328240de7ae842ac86e6351bf611e935 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "328240de7ae842ac86e6351bf611e935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:34.056185 9597 consensus_queue.cc:1048] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:34.064283 9597 raft_consensus.cc:993] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad: : Instructing follower 0b70f2147b7a477bb94eedf1daf42b28 to start an election
I20260430 07:57:34.064782 9542 raft_consensus.cc:1081] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Signalling peer 0b70f2147b7a477bb94eedf1daf42b28 to start an election
I20260430 07:57:34.066845 9420 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "667ece5d610e4127b462b1148129eca7"
dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
from {username='slave'} at 127.0.0.1:40604
I20260430 07:57:34.067401 9420 raft_consensus.cc:493] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:34.067713 9420 raft_consensus.cc:3060] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.072772 9420 raft_consensus.cc:515] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.075169 9420 leader_election.cc:290] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 2 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:34.076121 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "667ece5d610e4127b462b1148129eca7" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:34.076287 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "667ece5d610e4127b462b1148129eca7" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:34.076733 9346 raft_consensus.cc:3055] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:34.076848 9494 raft_consensus.cc:3060] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.077107 9346 raft_consensus.cc:740] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:34.077953 9346 consensus_queue.cc:260] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.079625 9346 raft_consensus.cc:3060] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.084141 9494 raft_consensus.cc:2468] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 2.
I20260430 07:57:34.085584 9386 leader_election.cc:304] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters:
I20260430 07:57:34.085602 9346 raft_consensus.cc:2468] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 2.
I20260430 07:57:34.086359 9590 raft_consensus.cc:2804] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:34.086840 9590 raft_consensus.cc:697] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 LEADER]: Becoming Leader. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Running, Role: LEADER
I20260430 07:57:34.087553 9590 consensus_queue.cc:237] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.093853 9244 catalog_manager.cc:5671] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 reported cstate change: term changed from 1 to 2, leader changed from 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129) to 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "0b70f2147b7a477bb94eedf1daf42b28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:34.123315 9590 consensus_queue.cc:1048] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:34.134545 9589 consensus_queue.cc:1048] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:57:34.153213 9589 consensus_queue.cc:1048] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:34.166178 9624 consensus_queue.cc:1048] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:34.172803 9574 raft_consensus.cc:993] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935: : Instructing follower 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:34.173357 9574 raft_consensus.cc:1081] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Signalling peer 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:34.175999 9345 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:44542
I20260430 07:57:34.176570 9345 raft_consensus.cc:493] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:34.176940 9345 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.186278 9345 raft_consensus.cc:515] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.187999 9345 leader_election.cc:290] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:34.188802 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:34.189152 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:34.189460 9420 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.189751 9494 raft_consensus.cc:3055] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:34.190099 9494 raft_consensus.cc:740] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:34.190898 9494 consensus_queue.cc:260] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.192178 9494 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.196745 9420 raft_consensus.cc:2468] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:34.197973 9312 leader_election.cc:304] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:34.198773 9542 raft_consensus.cc:2804] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:34.199021 9494 raft_consensus.cc:2468] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:34.199178 9542 raft_consensus.cc:697] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:34.200070 9542 consensus_queue.cc:237] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.205991 9244 catalog_manager.cc:5671] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 1 to 2, leader changed from 328240de7ae842ac86e6351bf611e935 (127.8.119.131) to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 2 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:34.219856 9543 raft_consensus.cc:993] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935: : Instructing follower 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:34.220326 9574 raft_consensus.cc:1081] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Signalling peer 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:34.221928 9345 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "234d7020308741a78ee34213c6cca517"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:44542
I20260430 07:57:34.223915 9345 raft_consensus.cc:493] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:34.224617 9345 raft_consensus.cc:3060] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.229461 9345 raft_consensus.cc:515] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.231093 9345 leader_election.cc:290] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:34.232031 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "234d7020308741a78ee34213c6cca517" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:34.232244 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "234d7020308741a78ee34213c6cca517" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:34.232679 9420 raft_consensus.cc:3060] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.232817 9494 raft_consensus.cc:3055] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:34.233173 9494 raft_consensus.cc:740] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:34.233995 9494 consensus_queue.cc:260] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.235231 9494 raft_consensus.cc:3060] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:34.241083 9420 raft_consensus.cc:2468] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:34.242861 9494 raft_consensus.cc:2468] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:34.242759 9312 leader_election.cc:304] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:34.244452 9542 raft_consensus.cc:2804] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:34.244902 9542 raft_consensus.cc:697] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:34.246052 9542 consensus_queue.cc:237] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:34.252590 9244 catalog_manager.cc:5671] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 1 to 2, leader changed from 328240de7ae842ac86e6351bf611e935 (127.8.119.131) to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 2 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:34.398456 9494 raft_consensus.cc:1275] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.399783 9542 consensus_queue.cc:1048] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:34.408185 9420 raft_consensus.cc:1275] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.409811 9595 consensus_queue.cc:1048] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:34.417008 9345 raft_consensus.cc:1275] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Refusing update from remote peer 0b70f2147b7a477bb94eedf1daf42b28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.418551 9624 consensus_queue.cc:1048] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:34.428558 9494 raft_consensus.cc:1275] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Refusing update from remote peer 0b70f2147b7a477bb94eedf1daf42b28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.430842 9589 consensus_queue.cc:1048] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:34.442747 9345 raft_consensus.cc:1275] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Refusing update from remote peer 328240de7ae842ac86e6351bf611e935: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.444245 9574 consensus_queue.cc:1048] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:34.452682 9420 raft_consensus.cc:1275] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Refusing update from remote peer 328240de7ae842ac86e6351bf611e935: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.453941 9573 consensus_queue.cc:1048] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:34.513553 9420 raft_consensus.cc:1275] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.515338 9595 consensus_queue.cc:1048] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:57:34.524221 9494 raft_consensus.cc:1275] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.525811 9595 consensus_queue.cc:1048] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:34.625622 9345 raft_consensus.cc:1275] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Refusing update from remote peer 0b70f2147b7a477bb94eedf1daf42b28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.627063 9589 consensus_queue.cc:1048] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:34.634853 9494 raft_consensus.cc:1275] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Refusing update from remote peer 0b70f2147b7a477bb94eedf1daf42b28: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.636402 9589 consensus_queue.cc:1048] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:34.709482 9494 raft_consensus.cc:1275] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.710850 9595 consensus_queue.cc:1048] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:34.719607 9420 raft_consensus.cc:1275] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.721611 9596 consensus_queue.cc:1048] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:34.723192 9494 raft_consensus.cc:1275] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.725042 9596 consensus_queue.cc:1048] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:34.736250 9420 raft_consensus.cc:1275] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:34.738122 9542 consensus_queue.cc:1048] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:36.865192 8670 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20260430 07:57:36.866884 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:36.892015 9346 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
mode: GRACEFUL
new_leader_uuid: "328240de7ae842ac86e6351bf611e935"
from {username='slave'} at 127.0.0.1:44568
I20260430 07:57:36.892858 9346 raft_consensus.cc:606] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Received request to transfer leadership to TS 328240de7ae842ac86e6351bf611e935
I20260430 07:57:36.896517 9346 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
mode: GRACEFUL
new_leader_uuid: "328240de7ae842ac86e6351bf611e935"
from {username='slave'} at 127.0.0.1:44568
I20260430 07:57:36.897068 9346 raft_consensus.cc:606] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Received request to transfer leadership to TS 328240de7ae842ac86e6351bf611e935
I20260430 07:57:36.898267 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260430 07:57:36.898684 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:37.021703 9632 raft_consensus.cc:993] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad: : Instructing follower 328240de7ae842ac86e6351bf611e935 to start an election
I20260430 07:57:37.022248 9634 raft_consensus.cc:1081] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Signalling peer 328240de7ae842ac86e6351bf611e935 to start an election
I20260430 07:57:37.024012 9494 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
from {username='slave'} at 127.0.0.1:34238
I20260430 07:57:37.024595 9494 raft_consensus.cc:493] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:37.024837 9494 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:37.029925 9494 raft_consensus.cc:515] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:37.031723 9494 leader_election.cc:290] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 3 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:37.032773 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:37.033030 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:37.033411 9420 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:37.034241 9346 raft_consensus.cc:3055] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:37.034838 9346 raft_consensus.cc:740] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:37.035789 9346 consensus_queue.cc:260] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:37.036724 9346 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:37.038455 9420 raft_consensus.cc:2468] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 3.
I20260430 07:57:37.039595 9461 leader_election.cc:304] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters:
I20260430 07:57:37.040355 9631 raft_consensus.cc:2804] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:37.041604 9631 raft_consensus.cc:697] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Becoming Leader. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:37.042575 9346 raft_consensus.cc:2468] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 3.
I20260430 07:57:37.042544 9631 consensus_queue.cc:237] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:37.050350 9240 catalog_manager.cc:5671] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 reported cstate change: term changed from 2 to 3, leader changed from 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129) to 328240de7ae842ac86e6351bf611e935 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "328240de7ae842ac86e6351bf611e935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:37.124419 9634 raft_consensus.cc:993] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad: : Instructing follower 328240de7ae842ac86e6351bf611e935 to start an election
I20260430 07:57:37.124900 9634 raft_consensus.cc:1081] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Signalling peer 328240de7ae842ac86e6351bf611e935 to start an election
I20260430 07:57:37.126482 9494 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
from {username='slave'} at 127.0.0.1:34238
I20260430 07:57:37.127048 9494 raft_consensus.cc:493] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:37.127336 9494 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:37.132825 9494 raft_consensus.cc:515] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:37.134358 9494 leader_election.cc:290] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 3 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:37.135331 9346 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:37.135403 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:37.136018 9420 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:37.136013 9346 raft_consensus.cc:3055] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:37.136456 9346 raft_consensus.cc:740] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:37.137210 9346 consensus_queue.cc:260] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:37.138221 9346 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:37.140508 9420 raft_consensus.cc:2468] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 3.
I20260430 07:57:37.141650 9461 leader_election.cc:304] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters:
I20260430 07:57:37.142375 9631 raft_consensus.cc:2804] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:37.142892 9631 raft_consensus.cc:697] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Becoming Leader. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:37.143652 9346 raft_consensus.cc:2468] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 3.
I20260430 07:57:37.143890 9631 consensus_queue.cc:237] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:37.156451 9240 catalog_manager.cc:5671] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 reported cstate change: term changed from 2 to 3, leader changed from 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129) to 328240de7ae842ac86e6351bf611e935 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "328240de7ae842ac86e6351bf611e935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:37.578456 9346 raft_consensus.cc:1275] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Refusing update from remote peer 328240de7ae842ac86e6351bf611e935: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:37.579741 9631 consensus_queue.cc:1048] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:37.591145 9420 raft_consensus.cc:1275] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Refusing update from remote peer 328240de7ae842ac86e6351bf611e935: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:37.592482 9649 consensus_queue.cc:1048] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:37.620333 9420 raft_consensus.cc:1275] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Refusing update from remote peer 328240de7ae842ac86e6351bf611e935: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:37.621714 9649 consensus_queue.cc:1048] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:37.634557 9346 raft_consensus.cc:1275] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Refusing update from remote peer 328240de7ae842ac86e6351bf611e935: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:37.636427 9638 consensus_queue.cc:1048] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:37.899713 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:37.906704 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:37.907186 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:38.908236 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:38.914357 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:38.914737 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:39.915962 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:39.922081 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:39.922640 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:40.924120 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:40.929497 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:40.929978 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:41.931069 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:41.937889 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:41.938275 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:42.939258 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:42.945320 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:42.945729 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:43.947353 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:43.953682 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:43.954151 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:44.955114 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:44.960812 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:44.961366 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:45.962517 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:45.968756 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:45.969347 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:47.004856 8670 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20260430 07:57:47.013072 9494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
mode: GRACEFUL
new_leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:34246
I20260430 07:57:47.013785 9494 raft_consensus.cc:606] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Received request to transfer leadership to TS 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:47.025203 9494 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
mode: GRACEFUL
new_leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:34246
I20260430 07:57:47.025774 9494 raft_consensus.cc:606] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Received request to transfer leadership to TS 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:47.049783 9421 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "667ece5d610e4127b462b1148129eca7"
dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
mode: GRACEFUL
new_leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:42910
I20260430 07:57:47.050316 9421 raft_consensus.cc:606] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 LEADER]: Received request to transfer leadership to TS 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:47.055773 9421 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4"
dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
mode: GRACEFUL
new_leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:42910
I20260430 07:57:47.056267 9421 raft_consensus.cc:606] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 LEADER]: Received request to transfer leadership to TS 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:47.061224 9421 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44351546bacf443ab17da51749a8ba47"
dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
mode: GRACEFUL
new_leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:42910
I20260430 07:57:47.062355 9421 raft_consensus.cc:606] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 LEADER]: Received request to transfer leadership to TS 48cc0d3c7a74492b9f815c91740b38ad
I20260430 07:57:47.199450 9665 raft_consensus.cc:993] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28: : Instructing follower 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:47.200103 9663 raft_consensus.cc:1081] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 LEADER]: Signalling peer 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:47.201905 9345 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:44528
I20260430 07:57:47.202466 9345 raft_consensus.cc:493] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:47.202836 9345 raft_consensus.cc:3060] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:47.209168 9345 raft_consensus.cc:515] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.211443 9345 leader_election.cc:290] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:47.212605 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:47.212642 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:47.213305 9421 raft_consensus.cc:3055] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:47.213449 9494 raft_consensus.cc:3060] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:47.213660 9421 raft_consensus.cc:740] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Running, Role: LEADER
I20260430 07:57:47.214442 9421 consensus_queue.cc:260] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.215389 9421 raft_consensus.cc:3060] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:47.219273 9494 raft_consensus.cc:2468] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:47.220475 9311 leader_election.cc:304] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:47.221318 9421 raft_consensus.cc:2468] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:47.221508 9632 raft_consensus.cc:2804] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:47.222785 9632 raft_consensus.cc:697] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:47.223492 9632 consensus_queue.cc:237] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.230845 9240 catalog_manager.cc:5671] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 1 to 2, leader changed from 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130) to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 2 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:47.262826 9662 raft_consensus.cc:993] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935: : Instructing follower 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:47.264348 9662 raft_consensus.cc:1081] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Signalling peer 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:47.266105 9346 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:44542
I20260430 07:57:47.266626 9346 raft_consensus.cc:493] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:47.266948 9346 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:47.267556 9675 raft_consensus.cc:993] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935: : Instructing follower 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:47.267982 9662 raft_consensus.cc:1081] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Signalling peer 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:47.269447 9345 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:44542
I20260430 07:57:47.270066 9345 raft_consensus.cc:493] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:47.270430 9345 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:47.274456 9346 raft_consensus.cc:515] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.276574 9346 leader_election.cc:290] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 4 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:47.277623 9494 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:47.277623 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6eda0175e14df28645991383e9f32c" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:47.278368 9494 raft_consensus.cc:3055] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:57:47.278543 9421 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:47.278739 9494 raft_consensus.cc:740] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:47.278520 9345 raft_consensus.cc:515] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.279631 9494 consensus_queue.cc:260] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.280748 9345 leader_election.cc:290] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 4 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:47.281045 9494 raft_consensus.cc:3060] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:47.281848 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:47.282222 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:47.282409 9420 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:47.282735 9495 raft_consensus.cc:3055] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:57:47.283059 9495 raft_consensus.cc:740] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:47.283927 9495 consensus_queue.cc:260] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.285121 9495 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:57:47.286796 9494 raft_consensus.cc:2468] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 4.
I20260430 07:57:47.287748 9311 leader_election.cc:304] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:47.288545 9632 raft_consensus.cc:2804] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:57:47.288997 9632 raft_consensus.cc:697] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 4 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:47.289824 9632 consensus_queue.cc:237] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.291041 9420 raft_consensus.cc:2468] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 4.
I20260430 07:57:47.292802 9312 leader_election.cc:304] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:47.294018 9669 raft_consensus.cc:2804] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:57:47.295751 9421 raft_consensus.cc:2468] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 4.
I20260430 07:57:47.302250 9495 raft_consensus.cc:2468] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 4.
I20260430 07:57:47.303876 9669 raft_consensus.cc:697] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 4 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:47.304929 9669 consensus_queue.cc:237] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.308740 9240 catalog_manager.cc:5671] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 3 to 4, leader changed from 328240de7ae842ac86e6351bf611e935 (127.8.119.131) to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 4 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:47.321009 9240 catalog_manager.cc:5671] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 3 to 4, leader changed from 328240de7ae842ac86e6351bf611e935 (127.8.119.131) to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 4 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:47.488238 9665 raft_consensus.cc:993] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28: : Instructing follower 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:47.488713 9663 raft_consensus.cc:1081] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 LEADER]: Signalling peer 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:47.490356 9345 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "667ece5d610e4127b462b1148129eca7"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:44528
I20260430 07:57:47.490883 9345 raft_consensus.cc:493] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:47.491156 9345 raft_consensus.cc:3060] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:47.496683 9345 raft_consensus.cc:515] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.498364 9345 leader_election.cc:290] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 3 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:47.499562 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "667ece5d610e4127b462b1148129eca7" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:47.499562 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "667ece5d610e4127b462b1148129eca7" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:47.500363 9421 raft_consensus.cc:3055] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:47.500360 9495 raft_consensus.cc:3060] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:47.500846 9421 raft_consensus.cc:740] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Running, Role: LEADER
I20260430 07:57:47.501657 9421 consensus_queue.cc:260] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.502832 9421 raft_consensus.cc:3060] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:47.507783 9495 raft_consensus.cc:2468] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 3.
I20260430 07:57:47.509756 9311 leader_election.cc:304] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:47.509895 9421 raft_consensus.cc:2468] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 3.
I20260430 07:57:47.510761 9669 raft_consensus.cc:2804] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:47.511499 9669 raft_consensus.cc:697] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:47.512552 9669 consensus_queue.cc:237] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.519861 9240 catalog_manager.cc:5671] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 2 to 3, leader changed from 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130) to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 3 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:47.592409 9665 raft_consensus.cc:993] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28: : Instructing follower 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:47.592864 9663 raft_consensus.cc:1081] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 LEADER]: Signalling peer 48cc0d3c7a74492b9f815c91740b38ad to start an election
I20260430 07:57:47.594460 9345 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "44351546bacf443ab17da51749a8ba47"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
from {username='slave'} at 127.0.0.1:44528
I20260430 07:57:47.595016 9345 raft_consensus.cc:493] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:47.595295 9345 raft_consensus.cc:3060] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:47.600075 9345 raft_consensus.cc:515] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.601974 9345 leader_election.cc:290] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Requested vote from peers 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:47.602825 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:47.602977 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "48cc0d3c7a74492b9f815c91740b38ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:47.603510 9421 raft_consensus.cc:3055] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:47.603595 9495 raft_consensus.cc:3060] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:47.603996 9421 raft_consensus.cc:740] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Running, Role: LEADER
I20260430 07:57:47.604856 9421 consensus_queue.cc:260] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.606202 9421 raft_consensus.cc:3060] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:47.610824 9495 raft_consensus.cc:2468] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:47.612417 9421 raft_consensus.cc:2468] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48cc0d3c7a74492b9f815c91740b38ad in term 2.
I20260430 07:57:47.612298 9311 leader_election.cc:304] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:47.613540 9669 raft_consensus.cc:2804] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:47.614583 9669 raft_consensus.cc:697] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Becoming Leader. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:47.615410 9669 consensus_queue.cc:237] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:47.620772 9421 raft_consensus.cc:1275] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:47.622300 9669 consensus_queue.cc:1048] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:47.623431 9240 catalog_manager.cc:5671] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad reported cstate change: term changed from 1 to 2, leader changed from 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130) to 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129). New cstate: current_term: 2 leader_uuid: "48cc0d3c7a74492b9f815c91740b38ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:47.640666 9495 raft_consensus.cc:1275] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:47.642346 9678 consensus_queue.cc:1048] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:47.830930 9495 raft_consensus.cc:1275] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 4 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260430 07:57:47.832376 9678 consensus_queue.cc:1048] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:57:47.842062 9421 raft_consensus.cc:1275] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 4 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260430 07:57:47.844486 9678 consensus_queue.cc:1048] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:57:47.879088 9420 raft_consensus.cc:1275] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 4 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260430 07:57:47.880461 9678 consensus_queue.cc:1048] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:57:47.891623 9495 raft_consensus.cc:1275] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 4 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260430 07:57:47.893973 9678 consensus_queue.cc:1048] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:57:47.938248 9495 raft_consensus.cc:1275] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:47.939685 9678 consensus_queue.cc:1048] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:47.952131 9420 raft_consensus.cc:1275] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:47.953549 9678 consensus_queue.cc:1048] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:48.043788 9495 raft_consensus.cc:1275] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:48.045472 9678 consensus_queue.cc:1048] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:48.056700 9420 raft_consensus.cc:1275] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Refusing update from remote peer 48cc0d3c7a74492b9f815c91740b38ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:48.058799 9678 consensus_queue.cc:1048] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:50.117235 8670 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20260430 07:57:50.118690 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:50.134337 9345 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
mode: GRACEFUL
new_leader_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
from {username='slave'} at 127.0.0.1:44568
I20260430 07:57:50.135632 9345 raft_consensus.cc:606] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Received request to transfer leadership to TS 0b70f2147b7a477bb94eedf1daf42b28
I20260430 07:57:50.138720 9345 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44351546bacf443ab17da51749a8ba47"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
mode: GRACEFUL
new_leader_uuid: "328240de7ae842ac86e6351bf611e935"
from {username='slave'} at 127.0.0.1:44568
I20260430 07:57:50.139230 9345 raft_consensus.cc:606] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Received request to transfer leadership to TS 328240de7ae842ac86e6351bf611e935
I20260430 07:57:50.141961 9345 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
mode: GRACEFUL
new_leader_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
from {username='slave'} at 127.0.0.1:44568
I20260430 07:57:50.142935 9345 raft_consensus.cc:606] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 4 LEADER]: Received request to transfer leadership to TS 0b70f2147b7a477bb94eedf1daf42b28
I20260430 07:57:50.146916 9345 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b6fa7604ad9b4a1a8e3323e6798f26ac"
dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
mode: GRACEFUL
new_leader_uuid: "328240de7ae842ac86e6351bf611e935"
from {username='slave'} at 127.0.0.1:44568
I20260430 07:57:50.147504 9345 raft_consensus.cc:606] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Received request to transfer leadership to TS 328240de7ae842ac86e6351bf611e935
I20260430 07:57:50.150179 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260430 07:57:50.151063 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:50.306843 9696 raft_consensus.cc:993] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad: : Instructing follower 328240de7ae842ac86e6351bf611e935 to start an election
I20260430 07:57:50.307703 9696 raft_consensus.cc:1081] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Signalling peer 328240de7ae842ac86e6351bf611e935 to start an election
I20260430 07:57:50.309680 9495 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b6fa7604ad9b4a1a8e3323e6798f26ac"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
from {username='slave'} at 127.0.0.1:34238
I20260430 07:57:50.310295 9495 raft_consensus.cc:493] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:50.310678 9495 raft_consensus.cc:3060] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:50.316884 9495 raft_consensus.cc:515] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:50.319906 9495 leader_election.cc:290] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 2 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:50.320528 9345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6fa7604ad9b4a1a8e3323e6798f26ac" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:50.321235 9421 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6fa7604ad9b4a1a8e3323e6798f26ac" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:50.321444 9345 raft_consensus.cc:3055] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:57:50.321823 9345 raft_consensus.cc:740] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:50.321846 9421 raft_consensus.cc:3060] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:50.322984 9345 consensus_queue.cc:260] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:50.324158 9345 raft_consensus.cc:3060] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:57:50.330166 9345 raft_consensus.cc:2468] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 2.
I20260430 07:57:50.331960 9421 raft_consensus.cc:2468] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 2.
I20260430 07:57:50.332027 9460 leader_election.cc:304] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 328240de7ae842ac86e6351bf611e935, 48cc0d3c7a74492b9f815c91740b38ad; no voters:
I20260430 07:57:50.335011 9698 raft_consensus.cc:2804] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:57:50.336032 9698 raft_consensus.cc:697] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 2 LEADER]: Becoming Leader. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:50.337419 9698 consensus_queue.cc:237] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } }
I20260430 07:57:50.352990 9240 catalog_manager.cc:5671] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 reported cstate change: term changed from 1 to 2, leader changed from 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129) to 328240de7ae842ac86e6351bf611e935 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "328240de7ae842ac86e6351bf611e935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:50.397846 9669 raft_consensus.cc:993] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad: : Instructing follower 0b70f2147b7a477bb94eedf1daf42b28 to start an election
I20260430 07:57:50.398288 9696 raft_consensus.cc:1081] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 4 LEADER]: Signalling peer 0b70f2147b7a477bb94eedf1daf42b28 to start an election
I20260430 07:57:50.399803 9421 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a"
dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
from {username='slave'} at 127.0.0.1:40604
I20260430 07:57:50.400264 9421 raft_consensus.cc:493] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:50.400519 9421 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:57:50.406481 9421 raft_consensus.cc:515] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:50.409204 9421 leader_election.cc:290] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 5 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:50.410303 9345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:50.410943 9345 raft_consensus.cc:3055] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:57:50.410990 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b4a4aefee4a479e8b197c722b7ee02a" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:50.411345 9345 raft_consensus.cc:740] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:50.411531 9495 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:57:50.412485 9345 consensus_queue.cc:260] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:50.414142 9345 raft_consensus.cc:3060] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:57:50.419255 9495 raft_consensus.cc:2468] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 5.
I20260430 07:57:50.421264 9386 leader_election.cc:304] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters:
I20260430 07:57:50.421653 9345 raft_consensus.cc:2468] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 5.
I20260430 07:57:50.422736 9663 raft_consensus.cc:2804] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:57:50.424166 9663 raft_consensus.cc:697] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 5 LEADER]: Becoming Leader. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Running, Role: LEADER
I20260430 07:57:50.425881 9663 consensus_queue.cc:237] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:50.435449 9240 catalog_manager.cc:5671] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 reported cstate change: term changed from 4 to 5, leader changed from 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129) to 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130). New cstate: current_term: 5 leader_uuid: "0b70f2147b7a477bb94eedf1daf42b28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:50.539722 9697 raft_consensus.cc:993] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad: : Instructing follower 0b70f2147b7a477bb94eedf1daf42b28 to start an election
I20260430 07:57:50.540261 9697 raft_consensus.cc:1081] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Signalling peer 0b70f2147b7a477bb94eedf1daf42b28 to start an election
I20260430 07:57:50.541976 9420 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4"
dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
from {username='slave'} at 127.0.0.1:40604
I20260430 07:57:50.542462 9420 raft_consensus.cc:493] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:50.542745 9420 raft_consensus.cc:3060] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:50.547931 9420 raft_consensus.cc:515] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:50.549433 9420 leader_election.cc:290] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 3 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 328240de7ae842ac86e6351bf611e935 (127.8.119.131:44085)
I20260430 07:57:50.550729 9345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:50.550985 9495 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21e1ae9826db429dbb15987b8f56dde4" candidate_uuid: "0b70f2147b7a477bb94eedf1daf42b28" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "328240de7ae842ac86e6351bf611e935"
I20260430 07:57:50.551316 9345 raft_consensus.cc:3055] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:50.551443 9495 raft_consensus.cc:3060] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:50.551653 9345 raft_consensus.cc:740] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:50.552500 9345 consensus_queue.cc:260] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:50.553799 9345 raft_consensus.cc:3060] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:50.558224 9345 raft_consensus.cc:2468] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 3.
I20260430 07:57:50.558472 9495 raft_consensus.cc:2468] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b70f2147b7a477bb94eedf1daf42b28 in term 3.
I20260430 07:57:50.559459 9386 leader_election.cc:304] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters:
I20260430 07:57:50.560096 9663 raft_consensus.cc:2804] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:50.560573 9663 raft_consensus.cc:697] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 LEADER]: Becoming Leader. State: Replica: 0b70f2147b7a477bb94eedf1daf42b28, State: Running, Role: LEADER
I20260430 07:57:50.561769 9663 consensus_queue.cc:237] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:50.570291 9240 catalog_manager.cc:5671] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 reported cstate change: term changed from 2 to 3, leader changed from 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129) to 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130). New cstate: current_term: 3 leader_uuid: "0b70f2147b7a477bb94eedf1daf42b28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: UNKNOWN } } }
I20260430 07:57:50.584465 9697 raft_consensus.cc:993] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad: : Instructing follower 328240de7ae842ac86e6351bf611e935 to start an election
I20260430 07:57:50.584992 9697 raft_consensus.cc:1081] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Signalling peer 328240de7ae842ac86e6351bf611e935 to start an election
I20260430 07:57:50.587168 9495 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "44351546bacf443ab17da51749a8ba47"
dest_uuid: "328240de7ae842ac86e6351bf611e935"
from {username='slave'} at 127.0.0.1:34238
I20260430 07:57:50.587677 9495 raft_consensus.cc:493] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:57:50.588050 9495 raft_consensus.cc:3060] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:50.593092 9495 raft_consensus.cc:515] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:50.594653 9495 leader_election.cc:290] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 3 election: Requested vote from peers 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037), 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267)
I20260430 07:57:50.595696 9345 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "48cc0d3c7a74492b9f815c91740b38ad"
I20260430 07:57:50.595696 9420 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44351546bacf443ab17da51749a8ba47" candidate_uuid: "328240de7ae842ac86e6351bf611e935" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0b70f2147b7a477bb94eedf1daf42b28"
I20260430 07:57:50.596303 9345 raft_consensus.cc:3055] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:57:50.596618 9420 raft_consensus.cc:3060] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:50.596635 9345 raft_consensus.cc:740] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48cc0d3c7a74492b9f815c91740b38ad, State: Running, Role: LEADER
I20260430 07:57:50.597488 9345 consensus_queue.cc:260] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:50.598361 9345 raft_consensus.cc:3060] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:57:50.601104 9420 raft_consensus.cc:2468] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 3.
I20260430 07:57:50.602471 9461 leader_election.cc:304] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b70f2147b7a477bb94eedf1daf42b28, 328240de7ae842ac86e6351bf611e935; no voters:
I20260430 07:57:50.602866 9345 raft_consensus.cc:2468] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 328240de7ae842ac86e6351bf611e935 in term 3.
I20260430 07:57:50.603469 9698 raft_consensus.cc:2804] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:57:50.604115 9698 raft_consensus.cc:697] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Becoming Leader. State: Replica: 328240de7ae842ac86e6351bf611e935, State: Running, Role: LEADER
I20260430 07:57:50.604997 9698 consensus_queue.cc:237] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } }
I20260430 07:57:50.614364 9240 catalog_manager.cc:5671] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 reported cstate change: term changed from 2 to 3, leader changed from 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129) to 328240de7ae842ac86e6351bf611e935 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "328240de7ae842ac86e6351bf611e935" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 } health_report { overall_health: HEALTHY } } }
I20260430 07:57:50.737742 9345 raft_consensus.cc:1275] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Refusing update from remote peer 328240de7ae842ac86e6351bf611e935: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:50.738988 9698 consensus_queue.cc:1048] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:57:50.750353 9420 raft_consensus.cc:1275] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Refusing update from remote peer 328240de7ae842ac86e6351bf611e935: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:57:50.752012 9712 consensus_queue.cc:1048] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:57:50.909554 9345 raft_consensus.cc:1275] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 5 FOLLOWER]: Refusing update from remote peer 0b70f2147b7a477bb94eedf1daf42b28: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260430 07:57:50.911746 9663 consensus_queue.cc:1048] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:57:50.923547 9495 raft_consensus.cc:1275] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 5 FOLLOWER]: Refusing update from remote peer 0b70f2147b7a477bb94eedf1daf42b28: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260430 07:57:50.925863 9663 consensus_queue.cc:1048] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:57:50.963055 9495 raft_consensus.cc:1275] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Refusing update from remote peer 0b70f2147b7a477bb94eedf1daf42b28: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:50.964572 9716 consensus_queue.cc:1048] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "328240de7ae842ac86e6351bf611e935" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 44085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:57:50.974958 9345 raft_consensus.cc:1275] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Refusing update from remote peer 0b70f2147b7a477bb94eedf1daf42b28: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:50.976795 9716 consensus_queue.cc:1048] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:51.119895 9345 raft_consensus.cc:1275] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Refusing update from remote peer 328240de7ae842ac86e6351bf611e935: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:51.121372 9712 consensus_queue.cc:1048] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48cc0d3c7a74492b9f815c91740b38ad" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:51.132130 9420 raft_consensus.cc:1275] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Refusing update from remote peer 328240de7ae842ac86e6351bf611e935: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:57:51.133728 9712 consensus_queue.cc:1048] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b70f2147b7a477bb94eedf1daf42b28" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:57:51.152737 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:51.158949 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:51.159438 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:52.160723 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:52.166813 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:52.167469 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:53.168642 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:53.176527 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:53.177057 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:54.177977 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:54.183902 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:54.184370 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:55.185433 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:55.191232 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:55.191778 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:56.192752 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:56.199270 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:56.199712 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:57.200601 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:57.207386 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:57.207868 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:58.208817 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:58.215827 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:58.216324 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:57:59.217492 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:57:59.222844 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260430 07:57:59.223357 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:58:00.266376 8670 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20260430 07:58:00.267064 8670 tablet_server.cc:179] TabletServer@127.8.119.129:0 shutting down...
W20260430 07:58:00.298815 9386 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20260430 07:58:00.308624 9386 consensus_peers.cc:597] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 -> Peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Couldn't send request to peer 48cc0d3c7a74492b9f815c91740b38ad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:00.312171 8670 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260430 07:58:00.313517 8670 tablet_replica.cc:333] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad: stopping tablet replica
I20260430 07:58:00.314340 8670 raft_consensus.cc:2243] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.314920 8670 raft_consensus.cc:2272] T 21e1ae9826db429dbb15987b8f56dde4 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.317237 8670 tablet_replica.cc:333] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad: stopping tablet replica
I20260430 07:58:00.318064 8670 raft_consensus.cc:2243] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.319336 8670 raft_consensus.cc:2272] T 234d7020308741a78ee34213c6cca517 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.322147 8670 tablet_replica.cc:333] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad: stopping tablet replica
W20260430 07:58:00.322456 9386 consensus_peers.cc:597] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 -> Peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Couldn't send request to peer 48cc0d3c7a74492b9f815c91740b38ad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:00.323040 8670 raft_consensus.cc:2243] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.324056 8670 raft_consensus.cc:2272] T 667ece5d610e4127b462b1148129eca7 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.326277 8670 tablet_replica.cc:333] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad: stopping tablet replica
I20260430 07:58:00.326812 8670 raft_consensus.cc:2243] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 5 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.327284 8670 raft_consensus.cc:2272] T 7b4a4aefee4a479e8b197c722b7ee02a P 48cc0d3c7a74492b9f815c91740b38ad [term 5 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.330065 8670 tablet_replica.cc:333] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad: stopping tablet replica
I20260430 07:58:00.331046 8670 raft_consensus.cc:2243] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.334049 8670 raft_consensus.cc:2272] T 904b5383e0484763a22dffe232eae9e9 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Raft consensus is shut down!
W20260430 07:58:00.335743 9460 consensus_peers.cc:597] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 -> Peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Couldn't send request to peer 48cc0d3c7a74492b9f815c91740b38ad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:00.337328 8670 tablet_replica.cc:333] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad: stopping tablet replica
I20260430 07:58:00.338007 8670 raft_consensus.cc:2243] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 4 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.338997 8670 raft_consensus.cc:2272] T 7c6eda0175e14df28645991383e9f32c P 48cc0d3c7a74492b9f815c91740b38ad [term 4 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.341142 8670 tablet_replica.cc:333] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad: stopping tablet replica
I20260430 07:58:00.341837 8670 raft_consensus.cc:2243] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.342350 8670 raft_consensus.cc:2272] T 44351546bacf443ab17da51749a8ba47 P 48cc0d3c7a74492b9f815c91740b38ad [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.344492 8670 tablet_replica.cc:333] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad: stopping tablet replica
I20260430 07:58:00.345624 8670 raft_consensus.cc:2243] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.346669 8670 raft_consensus.cc:2272] T b6fa7604ad9b4a1a8e3323e6798f26ac P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.349009 8670 tablet_replica.cc:333] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad: stopping tablet replica
I20260430 07:58:00.349965 8670 raft_consensus.cc:2243] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.350579 8670 raft_consensus.cc:2272] T fb2a4249b52b437387881cca082c5d17 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.352880 8670 tablet_replica.cc:333] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad: stopping tablet replica
I20260430 07:58:00.353462 8670 raft_consensus.cc:2243] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.353945 8670 raft_consensus.cc:2272] T 14d40b1b5ff64e3495088699c5874c42 P 48cc0d3c7a74492b9f815c91740b38ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.399214 8670 tablet_server.cc:196] TabletServer@127.8.119.129:0 shutdown complete.
I20260430 07:58:00.421885 8670 tablet_server.cc:179] TabletServer@127.8.119.130:0 shutting down...
I20260430 07:58:00.457099 8670 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260430 07:58:00.457849 8670 tablet_replica.cc:333] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28: stopping tablet replica
I20260430 07:58:00.458449 8670 raft_consensus.cc:2243] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.458951 8670 raft_consensus.cc:2272] T 44351546bacf443ab17da51749a8ba47 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.461452 8670 tablet_replica.cc:333] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28: stopping tablet replica
I20260430 07:58:00.462045 8670 raft_consensus.cc:2243] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.463117 8670 raft_consensus.cc:2272] T 14d40b1b5ff64e3495088699c5874c42 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.465221 8670 tablet_replica.cc:333] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28: stopping tablet replica
I20260430 07:58:00.465917 8670 raft_consensus.cc:2243] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260430 07:58:00.466104 9460 consensus_peers.cc:597] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 -> Peer 48cc0d3c7a74492b9f815c91740b38ad (127.8.119.129:34037): Couldn't send request to peer 48cc0d3c7a74492b9f815c91740b38ad. Status: Network error: Client connection negotiation failed: client connection to 127.8.119.129:34037: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:00.466368 8670 raft_consensus.cc:2272] T fb2a4249b52b437387881cca082c5d17 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.468322 8670 tablet_replica.cc:333] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28: stopping tablet replica
I20260430 07:58:00.469108 8670 raft_consensus.cc:2243] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.469643 8670 raft_consensus.cc:2272] T 7c6eda0175e14df28645991383e9f32c P 0b70f2147b7a477bb94eedf1daf42b28 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.472018 8670 tablet_replica.cc:333] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28: stopping tablet replica
I20260430 07:58:00.472654 8670 raft_consensus.cc:2243] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.473099 8670 raft_consensus.cc:2272] T 904b5383e0484763a22dffe232eae9e9 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.475193 8670 tablet_replica.cc:333] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28: stopping tablet replica
I20260430 07:58:00.475709 8670 raft_consensus.cc:2243] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.476119 8670 raft_consensus.cc:2272] T b6fa7604ad9b4a1a8e3323e6798f26ac P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.477924 8670 tablet_replica.cc:333] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28: stopping tablet replica
I20260430 07:58:00.478425 8670 raft_consensus.cc:2243] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.478868 8670 raft_consensus.cc:2272] T 667ece5d610e4127b462b1148129eca7 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.480604 8670 tablet_replica.cc:333] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28: stopping tablet replica
I20260430 07:58:00.481105 8670 raft_consensus.cc:2243] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 5 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.482394 8670 raft_consensus.cc:2272] T 7b4a4aefee4a479e8b197c722b7ee02a P 0b70f2147b7a477bb94eedf1daf42b28 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.484418 8670 tablet_replica.cc:333] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28: stopping tablet replica
I20260430 07:58:00.484988 8670 raft_consensus.cc:2243] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.486155 8670 raft_consensus.cc:2272] T 21e1ae9826db429dbb15987b8f56dde4 P 0b70f2147b7a477bb94eedf1daf42b28 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.488111 8670 tablet_replica.cc:333] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28: stopping tablet replica
I20260430 07:58:00.488606 8670 raft_consensus.cc:2243] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.489014 8670 raft_consensus.cc:2272] T 234d7020308741a78ee34213c6cca517 P 0b70f2147b7a477bb94eedf1daf42b28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.519944 8670 tablet_server.cc:196] TabletServer@127.8.119.130:0 shutdown complete.
I20260430 07:58:00.543403 8670 tablet_server.cc:179] TabletServer@127.8.119.131:0 shutting down...
W20260430 07:58:00.555317 9461 consensus_peers.cc:597] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 -> Peer 0b70f2147b7a477bb94eedf1daf42b28 (127.8.119.130:35267): Couldn't send request to peer 0b70f2147b7a477bb94eedf1daf42b28. Status: Network error: Client connection negotiation failed: client connection to 127.8.119.130:35267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:00.577067 8670 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260430 07:58:00.577979 8670 tablet_replica.cc:333] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935: stopping tablet replica
I20260430 07:58:00.578662 8670 raft_consensus.cc:2243] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 2 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.579499 8670 raft_consensus.cc:2272] T fb2a4249b52b437387881cca082c5d17 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.581555 8670 tablet_replica.cc:333] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935: stopping tablet replica
I20260430 07:58:00.582129 8670 raft_consensus.cc:2243] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.582585 8670 raft_consensus.cc:2272] T 904b5383e0484763a22dffe232eae9e9 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.584365 8670 tablet_replica.cc:333] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935: stopping tablet replica
I20260430 07:58:00.584895 8670 raft_consensus.cc:2243] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 3 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.586128 8670 raft_consensus.cc:2272] T 44351546bacf443ab17da51749a8ba47 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.587965 8670 tablet_replica.cc:333] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935: stopping tablet replica
I20260430 07:58:00.588469 8670 raft_consensus.cc:2243] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.588934 8670 raft_consensus.cc:2272] T 14d40b1b5ff64e3495088699c5874c42 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.591506 8670 tablet_replica.cc:333] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935: stopping tablet replica
I20260430 07:58:00.592136 8670 raft_consensus.cc:2243] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.592619 8670 raft_consensus.cc:2272] T 7c6eda0175e14df28645991383e9f32c P 328240de7ae842ac86e6351bf611e935 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.594439 8670 tablet_replica.cc:333] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935: stopping tablet replica
I20260430 07:58:00.595003 8670 raft_consensus.cc:2243] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 2 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.595799 8670 raft_consensus.cc:2272] T b6fa7604ad9b4a1a8e3323e6798f26ac P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.597715 8670 tablet_replica.cc:333] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935: stopping tablet replica
I20260430 07:58:00.598222 8670 raft_consensus.cc:2243] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.598675 8670 raft_consensus.cc:2272] T 21e1ae9826db429dbb15987b8f56dde4 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.600386 8670 tablet_replica.cc:333] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935: stopping tablet replica
I20260430 07:58:00.600884 8670 raft_consensus.cc:2243] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.601366 8670 raft_consensus.cc:2272] T 234d7020308741a78ee34213c6cca517 P 328240de7ae842ac86e6351bf611e935 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.602984 8670 tablet_replica.cc:333] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935: stopping tablet replica
I20260430 07:58:00.603452 8670 raft_consensus.cc:2243] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.603925 8670 raft_consensus.cc:2272] T 7b4a4aefee4a479e8b197c722b7ee02a P 328240de7ae842ac86e6351bf611e935 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.605666 8670 tablet_replica.cc:333] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935: stopping tablet replica
I20260430 07:58:00.606104 8670 raft_consensus.cc:2243] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:00.606510 8670 raft_consensus.cc:2272] T 667ece5d610e4127b462b1148129eca7 P 328240de7ae842ac86e6351bf611e935 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.637706 8670 tablet_server.cc:196] TabletServer@127.8.119.131:0 shutdown complete.
I20260430 07:58:00.661141 8670 master.cc:562] Master@127.8.119.190:46599 shutting down...
I20260430 07:58:00.693722 8670 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260430 07:58:00.694370 8670 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:00.694684 8670 tablet_replica.cc:333] T 00000000000000000000000000000000 P 1df1c13247cb4d6eb41c854c76db4fb2: stopping tablet replica
I20260430 07:58:00.714846 8670 master.cc:584] Master@127.8.119.190:46599 shutdown complete.
0b70f2147b7a477bb94eedf1daf42b28 4
328240de7ae842ac86e6351bf611e935 1
48cc0d3c7a74492b9f815c91740b38ad 5
0b70f2147b7a477bb94eedf1daf42b28 4
328240de7ae842ac86e6351bf611e935 3
48cc0d3c7a74492b9f815c91740b38ad 3
0b70f2147b7a477bb94eedf1daf42b28 1
328240de7ae842ac86e6351bf611e935 1
48cc0d3c7a74492b9f815c91740b38ad 8
0b70f2147b7a477bb94eedf1daf42b28 3
328240de7ae842ac86e6351bf611e935 3
48cc0d3c7a74492b9f815c91740b38ad 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28933 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260430 07:58:00.763907 8670 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.119.190:40689
I20260430 07:58:00.765754 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:58:00.772938 9736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:58:00.773083 9737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:58:00.774582 9739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:58:00.776101 8670 server_base.cc:1061] running on GCE node
I20260430 07:58:00.777060 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:58:00.777590 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:58:00.777772 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535880777748 us; error 0 us; skew 500 ppm
I20260430 07:58:00.778321 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:58:00.781440 8670 webserver.cc:492] Webserver started at http://127.8.119.190:45139/ using document root <none> and password file <none>
I20260430 07:58:00.782006 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:58:00.782198 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:58:00.782516 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:58:00.783762 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/master-0-root/instance:
uuid: "b5dc045545db48eba3c35052f59c6ced"
format_stamp: "Formatted at 2026-04-30 07:58:00 on dist-test-slave-6ct4"
I20260430 07:58:00.790442 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20260430 07:58:00.794147 9744 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:00.795194 8670 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260430 07:58:00.795449 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/master-0-root
uuid: "b5dc045545db48eba3c35052f59c6ced"
format_stamp: "Formatted at 2026-04-30 07:58:00 on dist-test-slave-6ct4"
I20260430 07:58:00.795711 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:58:00.808367 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:58:00.809696 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:58:00.859753 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.190:40689
I20260430 07:58:00.860069 9795 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.190:40689 every 8 connection(s)
I20260430 07:58:00.864483 9796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:00.875437 9796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced: Bootstrap starting.
I20260430 07:58:00.881187 9796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:00.886080 9796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced: No bootstrap required, opened a new log
I20260430 07:58:00.888345 9796 raft_consensus.cc:359] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc045545db48eba3c35052f59c6ced" member_type: VOTER }
I20260430 07:58:00.888809 9796 raft_consensus.cc:385] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:00.888994 9796 raft_consensus.cc:740] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dc045545db48eba3c35052f59c6ced, State: Initialized, Role: FOLLOWER
I20260430 07:58:00.889559 9796 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc045545db48eba3c35052f59c6ced" member_type: VOTER }
I20260430 07:58:00.890601 9796 raft_consensus.cc:399] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260430 07:58:00.891126 9796 raft_consensus.cc:493] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260430 07:58:00.891387 9796 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:00.897647 9796 raft_consensus.cc:515] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc045545db48eba3c35052f59c6ced" member_type: VOTER }
I20260430 07:58:00.898206 9796 leader_election.cc:304] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5dc045545db48eba3c35052f59c6ced; no voters:
I20260430 07:58:00.899353 9796 leader_election.cc:290] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260430 07:58:00.899910 9799 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:00.902320 9799 raft_consensus.cc:697] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [term 1 LEADER]: Becoming Leader. State: Replica: b5dc045545db48eba3c35052f59c6ced, State: Running, Role: LEADER
I20260430 07:58:00.903285 9799 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc045545db48eba3c35052f59c6ced" member_type: VOTER }
I20260430 07:58:00.903928 9796 sys_catalog.cc:565] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [sys.catalog]: configured and running, proceeding with master startup.
I20260430 07:58:00.910238 9800 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5dc045545db48eba3c35052f59c6ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc045545db48eba3c35052f59c6ced" member_type: VOTER } }
I20260430 07:58:00.911092 9800 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [sys.catalog]: This master's current role is: LEADER
I20260430 07:58:00.913494 9805 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260430 07:58:00.913767 9801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5dc045545db48eba3c35052f59c6ced. Latest consensus state: current_term: 1 leader_uuid: "b5dc045545db48eba3c35052f59c6ced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5dc045545db48eba3c35052f59c6ced" member_type: VOTER } }
I20260430 07:58:00.914444 9801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced [sys.catalog]: This master's current role is: LEADER
I20260430 07:58:00.919054 9805 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260430 07:58:00.924664 8670 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260430 07:58:00.935305 9805 catalog_manager.cc:1357] Generated new cluster ID: 067c0904ca20470f9354072e93ca8b71
I20260430 07:58:00.935611 9805 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260430 07:58:00.957469 9805 catalog_manager.cc:1380] Generated new certificate authority record
I20260430 07:58:00.959430 9805 catalog_manager.cc:1514] Loading token signing keys...
I20260430 07:58:00.978518 9805 catalog_manager.cc:6044] T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced: Generated new TSK 0
I20260430 07:58:00.979125 9805 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260430 07:58:00.998131 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:58:01.005580 9817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:58:01.006496 9818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:58:01.010113 9820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:58:01.010118 8670 server_base.cc:1061] running on GCE node
I20260430 07:58:01.011647 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:58:01.012120 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:58:01.012382 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535881012363 us; error 0 us; skew 500 ppm
I20260430 07:58:01.013012 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:58:01.015664 8670 webserver.cc:492] Webserver started at http://127.8.119.129:42587/ using document root <none> and password file <none>
I20260430 07:58:01.016187 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:58:01.016497 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:58:01.016952 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:58:01.018245 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root/instance:
uuid: "27f97a330f374849be58a168318c1ec6"
format_stamp: "Formatted at 2026-04-30 07:58:01 on dist-test-slave-6ct4"
I20260430 07:58:01.023341 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260430 07:58:01.028409 9825 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:01.029495 8670 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 07:58:01.029906 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root
uuid: "27f97a330f374849be58a168318c1ec6"
format_stamp: "Formatted at 2026-04-30 07:58:01 on dist-test-slave-6ct4"
I20260430 07:58:01.030296 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:58:01.050357 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:58:01.051841 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:58:01.053683 8670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 07:58:01.056051 8670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 07:58:01.056250 8670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:01.056427 8670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 07:58:01.056588 8670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:01.104212 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.129:36679
I20260430 07:58:01.104272 9887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.129:36679 every 8 connection(s)
I20260430 07:58:01.109233 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:58:01.118990 9892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:58:01.120620 9893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:58:01.125149 9895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:58:01.126145 8670 server_base.cc:1061] running on GCE node
I20260430 07:58:01.128527 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:58:01.129081 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:58:01.129421 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535881129397 us; error 0 us; skew 500 ppm
I20260430 07:58:01.130110 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:58:01.130955 9888 heartbeater.cc:344] Connected to a master server at 127.8.119.190:40689
I20260430 07:58:01.131345 9888 heartbeater.cc:461] Registering TS with master...
I20260430 07:58:01.132160 9888 heartbeater.cc:507] Master 127.8.119.190:40689 requested a full tablet report, sending...
I20260430 07:58:01.133737 8670 webserver.cc:492] Webserver started at http://127.8.119.130:38681/ using document root <none> and password file <none>
I20260430 07:58:01.134676 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:58:01.134950 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:58:01.135059 9761 ts_manager.cc:194] Registered new tserver with Master: 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679)
I20260430 07:58:01.135224 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:58:01.136644 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-1-root/instance:
uuid: "77e19ab653f540339c1c613fad1451b5"
format_stamp: "Formatted at 2026-04-30 07:58:01 on dist-test-slave-6ct4"
I20260430 07:58:01.137558 9761 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48090
I20260430 07:58:01.141892 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260430 07:58:01.146804 9900 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:01.148818 8670 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260430 07:58:01.149421 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-1-root
uuid: "77e19ab653f540339c1c613fad1451b5"
format_stamp: "Formatted at 2026-04-30 07:58:01 on dist-test-slave-6ct4"
I20260430 07:58:01.149843 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:58:01.167156 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:58:01.168491 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:58:01.170259 8670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 07:58:01.172871 8670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 07:58:01.173154 8670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:01.173436 8670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 07:58:01.173615 8670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:01.216601 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.130:38147
I20260430 07:58:01.216758 9962 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.130:38147 every 8 connection(s)
I20260430 07:58:01.221675 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:58:01.230091 9966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:58:01.231740 9967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:58:01.237986 9969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:58:01.239089 8670 server_base.cc:1061] running on GCE node
I20260430 07:58:01.239697 9963 heartbeater.cc:344] Connected to a master server at 127.8.119.190:40689
I20260430 07:58:01.240206 9963 heartbeater.cc:461] Registering TS with master...
I20260430 07:58:01.240275 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:58:01.240623 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:58:01.240896 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535881240798 us; error 0 us; skew 500 ppm
I20260430 07:58:01.241664 9963 heartbeater.cc:507] Master 127.8.119.190:40689 requested a full tablet report, sending...
I20260430 07:58:01.242220 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:58:01.245036 9761 ts_manager.cc:194] Registered new tserver with Master: 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:01.245693 8670 webserver.cc:492] Webserver started at http://127.8.119.131:44747/ using document root <none> and password file <none>
I20260430 07:58:01.246168 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:58:01.246317 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:58:01.246521 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:58:01.246611 9761 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48094
I20260430 07:58:01.247838 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-2-root/instance:
uuid: "48e81934d52a4e93be5171ceea193032"
format_stamp: "Formatted at 2026-04-30 07:58:01 on dist-test-slave-6ct4"
I20260430 07:58:01.253252 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260430 07:58:01.257469 9974 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:01.258566 8670 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260430 07:58:01.259028 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-2-root
uuid: "48e81934d52a4e93be5171ceea193032"
format_stamp: "Formatted at 2026-04-30 07:58:01 on dist-test-slave-6ct4"
I20260430 07:58:01.259398 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:58:01.274482 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:58:01.275871 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:58:01.277661 8670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 07:58:01.279989 8670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 07:58:01.280241 8670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:01.280501 8670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 07:58:01.280690 8670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:01.332288 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.131:39205
I20260430 07:58:01.332424 10036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.131:39205 every 8 connection(s)
I20260430 07:58:01.349695 10037 heartbeater.cc:344] Connected to a master server at 127.8.119.190:40689
I20260430 07:58:01.350152 10037 heartbeater.cc:461] Registering TS with master...
I20260430 07:58:01.351068 10037 heartbeater.cc:507] Master 127.8.119.190:40689 requested a full tablet report, sending...
I20260430 07:58:01.353327 9761 ts_manager.cc:194] Registered new tserver with Master: 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:01.354339 8670 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018853987s
I20260430 07:58:01.355079 9761 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48098
I20260430 07:58:01.388305 9761 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48104:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260430 07:58:01.395012 9761 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260430 07:58:01.620849 9928 tablet_service.cc:1511] Processing CreateTablet for tablet e51196f9c4fb44bf8f6296f5394417bb (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260430 07:58:01.622443 9928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e51196f9c4fb44bf8f6296f5394417bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.627511 9853 tablet_service.cc:1511] Processing CreateTablet for tablet e51196f9c4fb44bf8f6296f5394417bb (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260430 07:58:01.628212 9852 tablet_service.cc:1511] Processing CreateTablet for tablet 7c393abc9c604fc9aff3cedf85f68725 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260430 07:58:01.629154 9853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e51196f9c4fb44bf8f6296f5394417bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.630561 9852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c393abc9c604fc9aff3cedf85f68725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.628784 9851 tablet_service.cc:1511] Processing CreateTablet for tablet db24b74e2fa44ae2beba675582424a3b (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260430 07:58:01.631680 9850 tablet_service.cc:1511] Processing CreateTablet for tablet a81d1c3211f94be6a77c0c0902142e68 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260430 07:58:01.634598 9926 tablet_service.cc:1511] Processing CreateTablet for tablet db24b74e2fa44ae2beba675582424a3b (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260430 07:58:01.636281 9926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db24b74e2fa44ae2beba675582424a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.637212 10002 tablet_service.cc:1511] Processing CreateTablet for tablet e51196f9c4fb44bf8f6296f5394417bb (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260430 07:58:01.638669 10002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e51196f9c4fb44bf8f6296f5394417bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.638484 9925 tablet_service.cc:1511] Processing CreateTablet for tablet a81d1c3211f94be6a77c0c0902142e68 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260430 07:58:01.638942 9847 tablet_service.cc:1511] Processing CreateTablet for tablet a66ec69e8f2c493fa90c40e25719d298 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260430 07:58:01.640975 9927 tablet_service.cc:1511] Processing CreateTablet for tablet 7c393abc9c604fc9aff3cedf85f68725 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260430 07:58:01.634768 9849 tablet_service.cc:1511] Processing CreateTablet for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260430 07:58:01.642827 9848 tablet_service.cc:1511] Processing CreateTablet for tablet bddad6955cd64efd850fddc61341be7e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260430 07:58:01.643141 9846 tablet_service.cc:1511] Processing CreateTablet for tablet e537f351c19b43fd8b0b0a89cf897487 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260430 07:58:01.646234 9924 tablet_service.cc:1511] Processing CreateTablet for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260430 07:58:01.647486 9845 tablet_service.cc:1511] Processing CreateTablet for tablet 9001555606564c6081e62a65ffa52dc8 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260430 07:58:01.650135 9844 tablet_service.cc:1511] Processing CreateTablet for tablet 1aa1e40099b3420eaada73851145a79e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260430 07:58:01.651324 9922 tablet_service.cc:1511] Processing CreateTablet for tablet a66ec69e8f2c493fa90c40e25719d298 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260430 07:58:01.654593 10001 tablet_service.cc:1511] Processing CreateTablet for tablet 7c393abc9c604fc9aff3cedf85f68725 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260430 07:58:01.655073 9923 tablet_service.cc:1511] Processing CreateTablet for tablet bddad6955cd64efd850fddc61341be7e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260430 07:58:01.656958 10001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c393abc9c604fc9aff3cedf85f68725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.631683 9851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db24b74e2fa44ae2beba675582424a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.659070 9849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.659811 9846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e537f351c19b43fd8b0b0a89cf897487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.661643 9847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66ec69e8f2c493fa90c40e25719d298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.663465 9848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bddad6955cd64efd850fddc61341be7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.669847 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9001555606564c6081e62a65ffa52dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.671844 9850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a81d1c3211f94be6a77c0c0902142e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.673812 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa1e40099b3420eaada73851145a79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.673311 10000 tablet_service.cc:1511] Processing CreateTablet for tablet db24b74e2fa44ae2beba675582424a3b (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260430 07:58:01.675024 10000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db24b74e2fa44ae2beba675582424a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.683802 9999 tablet_service.cc:1511] Processing CreateTablet for tablet a81d1c3211f94be6a77c0c0902142e68 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260430 07:58:01.685417 9999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a81d1c3211f94be6a77c0c0902142e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.693956 9925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a81d1c3211f94be6a77c0c0902142e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.695295 9998 tablet_service.cc:1511] Processing CreateTablet for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260430 07:58:01.696908 9998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.697829 9927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c393abc9c604fc9aff3cedf85f68725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.701856 9922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66ec69e8f2c493fa90c40e25719d298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.705829 9924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.707177 9997 tablet_service.cc:1511] Processing CreateTablet for tablet bddad6955cd64efd850fddc61341be7e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260430 07:58:01.708933 9997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bddad6955cd64efd850fddc61341be7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.710026 9923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bddad6955cd64efd850fddc61341be7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.716504 9996 tablet_service.cc:1511] Processing CreateTablet for tablet a66ec69e8f2c493fa90c40e25719d298 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260430 07:58:01.718070 9996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66ec69e8f2c493fa90c40e25719d298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.731736 9921 tablet_service.cc:1511] Processing CreateTablet for tablet e537f351c19b43fd8b0b0a89cf897487 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260430 07:58:01.734648 9921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e537f351c19b43fd8b0b0a89cf897487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.749950 9995 tablet_service.cc:1511] Processing CreateTablet for tablet e537f351c19b43fd8b0b0a89cf897487 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260430 07:58:01.752770 9995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e537f351c19b43fd8b0b0a89cf897487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.762951 9920 tablet_service.cc:1511] Processing CreateTablet for tablet 9001555606564c6081e62a65ffa52dc8 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260430 07:58:01.764566 9920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9001555606564c6081e62a65ffa52dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.764170 9994 tablet_service.cc:1511] Processing CreateTablet for tablet 9001555606564c6081e62a65ffa52dc8 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260430 07:58:01.765817 9994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9001555606564c6081e62a65ffa52dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.770228 9919 tablet_service.cc:1511] Processing CreateTablet for tablet 1aa1e40099b3420eaada73851145a79e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260430 07:58:01.770612 9993 tablet_service.cc:1511] Processing CreateTablet for tablet 1aa1e40099b3420eaada73851145a79e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260430 07:58:01.772064 9993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa1e40099b3420eaada73851145a79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.772083 9919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1aa1e40099b3420eaada73851145a79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.866889 9925 tablet_service.cc:1511] Processing CreateTablet for tablet f8946312605d455cbfe6546ad6873ec1 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260430 07:58:01.868824 9925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8946312605d455cbfe6546ad6873ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.891258 9998 tablet_service.cc:1511] Processing CreateTablet for tablet 1b60aa0987ed4a9185a3fe724e0acf23 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260430 07:58:01.892649 9998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b60aa0987ed4a9185a3fe724e0acf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.923427 9994 tablet_service.cc:1511] Processing CreateTablet for tablet 15f2cd3a47984929840836a88ecfd8b0 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
W20260430 07:58:01.924257 9809 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:58:01.924959 9994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f2cd3a47984929840836a88ecfd8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.930433 9849 tablet_service.cc:1511] Processing CreateTablet for tablet f8946312605d455cbfe6546ad6873ec1 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260430 07:58:01.932191 9849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8946312605d455cbfe6546ad6873ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.934126 9925 tablet_service.cc:1511] Processing CreateTablet for tablet 1b60aa0987ed4a9185a3fe724e0acf23 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260430 07:58:01.935488 9925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b60aa0987ed4a9185a3fe724e0acf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.880481 9847 tablet_service.cc:1511] Processing CreateTablet for tablet 1b60aa0987ed4a9185a3fe724e0acf23 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260430 07:58:01.940477 9847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b60aa0987ed4a9185a3fe724e0acf23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.942378 9922 tablet_service.cc:1511] Processing CreateTablet for tablet 15f2cd3a47984929840836a88ecfd8b0 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260430 07:58:01.954352 10052 tablet_bootstrap.cc:492] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:01.957850 9848 tablet_service.cc:1511] Processing CreateTablet for tablet 022dabb87c374f5c8ae73005ad2fdad6 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260430 07:58:01.967506 9923 tablet_service.cc:1511] Processing CreateTablet for tablet 022dabb87c374f5c8ae73005ad2fdad6 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260430 07:58:01.873030 9850 tablet_service.cc:1511] Processing CreateTablet for tablet 15f2cd3a47984929840836a88ecfd8b0 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260430 07:58:01.980577 9848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 022dabb87c374f5c8ae73005ad2fdad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.962914 9853 tablet_service.cc:1511] Processing CreateTablet for tablet 926617cd6f3149f6a70ccb92f4504659 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260430 07:58:01.965544 9922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f2cd3a47984929840836a88ecfd8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.984771 9923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 022dabb87c374f5c8ae73005ad2fdad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.961125 9852 tablet_service.cc:1511] Processing CreateTablet for tablet a48a2b75c3a34252bf57a4f634457236 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260430 07:58:01.981567 10001 tablet_service.cc:1511] Processing CreateTablet for tablet f8946312605d455cbfe6546ad6873ec1 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260430 07:58:01.991469 9850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15f2cd3a47984929840836a88ecfd8b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.992153 10054 tablet_bootstrap.cc:492] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:01.994004 9853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926617cd6f3149f6a70ccb92f4504659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.995972 9844 tablet_service.cc:1511] Processing CreateTablet for tablet 7f111c6c48d34a6fb38048804967ed93 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260430 07:58:01.997303 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f111c6c48d34a6fb38048804967ed93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.998401 10053 tablet_bootstrap.cc:492] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:02.002691 9927 tablet_service.cc:1511] Processing CreateTablet for tablet a48a2b75c3a34252bf57a4f634457236 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260430 07:58:02.002421 9997 tablet_service.cc:1511] Processing CreateTablet for tablet a48a2b75c3a34252bf57a4f634457236 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260430 07:58:02.004778 9997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48a2b75c3a34252bf57a4f634457236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.004570 9995 tablet_service.cc:1511] Processing CreateTablet for tablet 981fd927b4114a73b86c1e232f121c7f (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260430 07:58:02.005213 9852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48a2b75c3a34252bf57a4f634457236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.006513 9995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 981fd927b4114a73b86c1e232f121c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.019049 10001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8946312605d455cbfe6546ad6873ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.023029 9920 tablet_service.cc:1511] Processing CreateTablet for tablet 7f111c6c48d34a6fb38048804967ed93 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260430 07:58:02.031083 9999 tablet_service.cc:1511] Processing CreateTablet for tablet 926617cd6f3149f6a70ccb92f4504659 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260430 07:58:02.016386 9924 tablet_service.cc:1511] Processing CreateTablet for tablet 926617cd6f3149f6a70ccb92f4504659 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260430 07:58:02.033416 9999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926617cd6f3149f6a70ccb92f4504659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.004570 10002 tablet_service.cc:1511] Processing CreateTablet for tablet 7f111c6c48d34a6fb38048804967ed93 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260430 07:58:02.039736 9845 tablet_service.cc:1511] Processing CreateTablet for tablet 76d5ae1122e343f28f49a7fdbf3db843 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260430 07:58:02.059057 9846 tablet_service.cc:1511] Processing CreateTablet for tablet 1caba957d05d44788146efd70334d6fe (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260430 07:58:02.062764 9851 tablet_service.cc:1511] Processing CreateTablet for tablet 981fd927b4114a73b86c1e232f121c7f (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260430 07:58:02.041148 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76d5ae1122e343f28f49a7fdbf3db843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.070350 10053 tablet_bootstrap.cc:654] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:02.071635 9851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 981fd927b4114a73b86c1e232f121c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.072499 9996 tablet_service.cc:1511] Processing CreateTablet for tablet 1caba957d05d44788146efd70334d6fe (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260430 07:58:02.092236 9846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1caba957d05d44788146efd70334d6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:01.961120 10052 tablet_bootstrap.cc:654] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:02.079648 9998 tablet_service.cc:1511] Processing CreateTablet for tablet e18e051a7225424ebb64aedb21bb97df (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260430 07:58:02.036257 9994 tablet_service.cc:1511] Processing CreateTablet for tablet 022dabb87c374f5c8ae73005ad2fdad6 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260430 07:58:02.085868 9850 tablet_service.cc:1511] Processing CreateTablet for tablet e18e051a7225424ebb64aedb21bb97df (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260430 07:58:02.109876 9850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18e051a7225424ebb64aedb21bb97df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.112428 10000 tablet_service.cc:1511] Processing CreateTablet for tablet 76d5ae1122e343f28f49a7fdbf3db843 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260430 07:58:02.089745 10002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f111c6c48d34a6fb38048804967ed93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.098351 9927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48a2b75c3a34252bf57a4f634457236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.118335 9924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926617cd6f3149f6a70ccb92f4504659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.100570 9928 tablet_service.cc:1511] Processing CreateTablet for tablet 1caba957d05d44788146efd70334d6fe (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260430 07:58:02.100620 9921 tablet_service.cc:1511] Processing CreateTablet for tablet 76d5ae1122e343f28f49a7fdbf3db843 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260430 07:58:02.120409 9928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1caba957d05d44788146efd70334d6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.122438 9921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76d5ae1122e343f28f49a7fdbf3db843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.128934 9920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f111c6c48d34a6fb38048804967ed93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.103714 9919 tablet_service.cc:1511] Processing CreateTablet for tablet e18e051a7225424ebb64aedb21bb97df (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260430 07:58:02.132437 9919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18e051a7225424ebb64aedb21bb97df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.103580 9926 tablet_service.cc:1511] Processing CreateTablet for tablet 981fd927b4114a73b86c1e232f121c7f (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260430 07:58:02.113523 9925 tablet_service.cc:1511] Processing CreateTablet for tablet eaaca7c934d147a396bbaee7e10d6496 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260430 07:58:02.088362 9997 tablet_service.cc:1511] Processing CreateTablet for tablet eaaca7c934d147a396bbaee7e10d6496 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260430 07:58:02.136406 10054 tablet_bootstrap.cc:654] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:02.141173 9888 heartbeater.cc:499] Master 127.8.119.190:40689 was elected leader, sending a full tablet report...
I20260430 07:58:02.162317 9926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 981fd927b4114a73b86c1e232f121c7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.164716 9923 tablet_service.cc:1511] Processing CreateTablet for tablet c420e10ec8014b4f8df55f542d9693f3 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260430 07:58:02.166244 9923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c420e10ec8014b4f8df55f542d9693f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.148265 9996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1caba957d05d44788146efd70334d6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.178077 9997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaaca7c934d147a396bbaee7e10d6496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.178758 9847 tablet_service.cc:1511] Processing CreateTablet for tablet eaaca7c934d147a396bbaee7e10d6496 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260430 07:58:02.180781 9847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaaca7c934d147a396bbaee7e10d6496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.182474 9998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18e051a7225424ebb64aedb21bb97df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.191936 9848 tablet_service.cc:1511] Processing CreateTablet for tablet c420e10ec8014b4f8df55f542d9693f3 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260430 07:58:02.193447 9848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c420e10ec8014b4f8df55f542d9693f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.194373 9994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 022dabb87c374f5c8ae73005ad2fdad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.200554 9925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaaca7c934d147a396bbaee7e10d6496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.208081 9928 tablet_service.cc:1511] Processing CreateTablet for tablet 7303f2c98e1b411f83314a2c2f146d4b (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260430 07:58:02.227598 10000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76d5ae1122e343f28f49a7fdbf3db843. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.227741 9928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7303f2c98e1b411f83314a2c2f146d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.230634 9845 tablet_service.cc:1511] Processing CreateTablet for tablet 7303f2c98e1b411f83314a2c2f146d4b (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260430 07:58:02.237851 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7303f2c98e1b411f83314a2c2f146d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.249092 9963 heartbeater.cc:499] Master 127.8.119.190:40689 was elected leader, sending a full tablet report...
I20260430 07:58:02.273427 10052 tablet_bootstrap.cc:492] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:02.274025 10052 ts_tablet_manager.cc:1403] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.320s user 0.010s sys 0.013s
W20260430 07:58:02.277652 9747 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 70798.3us
W20260430 07:58:02.277865 9748 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 52951.7us
I20260430 07:58:02.288028 10053 tablet_bootstrap.cc:492] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:02.288580 10053 ts_tablet_manager.cc:1403] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.291s user 0.014s sys 0.007s
I20260430 07:58:02.290830 9844 tablet_service.cc:1511] Processing CreateTablet for tablet 452aa377ba5f47c3bdd4bf5d4e48059d (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260430 07:58:02.292351 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452aa377ba5f47c3bdd4bf5d4e48059d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.292454 9922 tablet_service.cc:1511] Processing CreateTablet for tablet 58ea2e276db24da3890f352cef28ff97 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260430 07:58:02.293769 9922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ea2e276db24da3890f352cef28ff97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.299612 9849 tablet_service.cc:1511] Processing CreateTablet for tablet 9fc86c93a0014c789fda84211b6bd437 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260430 07:58:02.301014 9849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc86c93a0014c789fda84211b6bd437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.300927 10052 raft_consensus.cc:359] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.328694 10052 raft_consensus.cc:385] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:02.329062 10052 raft_consensus.cc:740] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:02.329469 9993 tablet_service.cc:1511] Processing CreateTablet for tablet c420e10ec8014b4f8df55f542d9693f3 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260430 07:58:02.329803 10052 consensus_queue.cc:260] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.303697 10054 tablet_bootstrap.cc:492] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:02.290833 9853 tablet_service.cc:1511] Processing CreateTablet for tablet 58ea2e276db24da3890f352cef28ff97 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260430 07:58:02.331162 9993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c420e10ec8014b4f8df55f542d9693f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.331344 10054 ts_tablet_manager.cc:1403] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.340s user 0.019s sys 0.003s
I20260430 07:58:02.331808 9853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ea2e276db24da3890f352cef28ff97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.334389 10054 raft_consensus.cc:359] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.335156 10054 raft_consensus.cc:385] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:02.335464 10054 raft_consensus.cc:740] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:02.308244 10001 tablet_service.cc:1511] Processing CreateTablet for tablet 7303f2c98e1b411f83314a2c2f146d4b (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260430 07:58:02.311090 10053 raft_consensus.cc:359] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.344122 10053 raft_consensus.cc:385] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:02.344517 10053 raft_consensus.cc:740] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:02.344360 9925 tablet_service.cc:1511] Processing CreateTablet for tablet ac5d1b9983db48e198134c6ab5246fbb (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260430 07:58:02.311945 9923 tablet_service.cc:1511] Processing CreateTablet for tablet 452aa377ba5f47c3bdd4bf5d4e48059d (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260430 07:58:02.313975 9928 tablet_service.cc:1511] Processing CreateTablet for tablet 9fc86c93a0014c789fda84211b6bd437 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260430 07:58:02.346179 9928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc86c93a0014c789fda84211b6bd437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.346647 10053 consensus_queue.cc:260] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.320006 9997 tablet_service.cc:1511] Processing CreateTablet for tablet 58ea2e276db24da3890f352cef28ff97 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260430 07:58:02.359439 10037 heartbeater.cc:499] Master 127.8.119.190:40689 was elected leader, sending a full tablet report...
I20260430 07:58:02.363021 9919 tablet_service.cc:1511] Processing CreateTablet for tablet 43b558e289a64355972c2e915f633bb9 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260430 07:58:02.363857 10054 consensus_queue.cc:260] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.419552 10060 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:02.421430 9997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ea2e276db24da3890f352cef28ff97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.421608 10053 ts_tablet_manager.cc:1434] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.133s user 0.010s sys 0.005s
I20260430 07:58:02.430971 10001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7303f2c98e1b411f83314a2c2f146d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.438225 9993 tablet_service.cc:1511] Processing CreateTablet for tablet 5cfb45b2ec3c45c38f2aba832ed29742 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260430 07:58:02.439674 9993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cfb45b2ec3c45c38f2aba832ed29742. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.439515 9994 tablet_service.cc:1511] Processing CreateTablet for tablet b978c5a9ab41402c806352f5ace14295 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260430 07:58:02.440974 9994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b978c5a9ab41402c806352f5ace14295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.423669 9852 tablet_service.cc:1511] Processing CreateTablet for tablet 43b558e289a64355972c2e915f633bb9 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260430 07:58:02.425876 9995 tablet_service.cc:1511] Processing CreateTablet for tablet 452aa377ba5f47c3bdd4bf5d4e48059d (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260430 07:58:02.443754 9849 tablet_service.cc:1511] Processing CreateTablet for tablet 5cfb45b2ec3c45c38f2aba832ed29742 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260430 07:58:02.444885 9995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452aa377ba5f47c3bdd4bf5d4e48059d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.444947 9849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cfb45b2ec3c45c38f2aba832ed29742. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.425876 10000 tablet_service.cc:1511] Processing CreateTablet for tablet 9fc86c93a0014c789fda84211b6bd437 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260430 07:58:02.427445 10002 tablet_service.cc:1511] Processing CreateTablet for tablet 43b558e289a64355972c2e915f633bb9 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260430 07:58:02.447108 9846 tablet_service.cc:1511] Processing CreateTablet for tablet b978c5a9ab41402c806352f5ace14295 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260430 07:58:02.447562 10000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc86c93a0014c789fda84211b6bd437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.448177 10002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b558e289a64355972c2e915f633bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.448329 9846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b978c5a9ab41402c806352f5ace14295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.453575 10060 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.455010 9852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b558e289a64355972c2e915f633bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.397943 9925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5d1b9983db48e198134c6ab5246fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.415592 9845 tablet_service.cc:1511] Processing CreateTablet for tablet ac5d1b9983db48e198134c6ab5246fbb (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260430 07:58:02.471947 9919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43b558e289a64355972c2e915f633bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.475121 10053 tablet_bootstrap.cc:492] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:02.475678 9920 tablet_service.cc:1511] Processing CreateTablet for tablet 5cfb45b2ec3c45c38f2aba832ed29742 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260430 07:58:02.478343 9927 tablet_service.cc:1511] Processing CreateTablet for tablet b978c5a9ab41402c806352f5ace14295 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260430 07:58:02.481531 9924 tablet_service.cc:1511] Processing CreateTablet for tablet de66bf2ed62e49aa95b1deff0383d037 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260430 07:58:02.482127 10053 tablet_bootstrap.cc:654] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:02.442431 9999 tablet_service.cc:1511] Processing CreateTablet for tablet 18a52542f8aa4aeca8e319d4f60e9699 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260430 07:58:02.503759 9920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cfb45b2ec3c45c38f2aba832ed29742. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.497771 10060 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:02.519308 10058 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:02.520215 10058 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.521153 10054 ts_tablet_manager.cc:1434] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.189s user 0.012s sys 0.025s
I20260430 07:58:02.537936 9998 tablet_service.cc:1511] Processing CreateTablet for tablet ac5d1b9983db48e198134c6ab5246fbb (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260430 07:58:02.534701 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5d1b9983db48e198134c6ab5246fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.543116 9847 tablet_service.cc:1511] Processing CreateTablet for tablet 18a52542f8aa4aeca8e319d4f60e9699 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260430 07:58:02.510741 9844 tablet_service.cc:1511] Processing CreateTablet for tablet de66bf2ed62e49aa95b1deff0383d037 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260430 07:58:02.545472 10052 ts_tablet_manager.cc:1434] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.271s user 0.031s sys 0.007s
I20260430 07:58:02.545712 9851 tablet_service.cc:1511] Processing CreateTablet for tablet 9c37b1d9d0b54e12b8ce6cd28f1fdb07 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260430 07:58:02.550467 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de66bf2ed62e49aa95b1deff0383d037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.551441 9999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a52542f8aa4aeca8e319d4f60e9699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.556692 9923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452aa377ba5f47c3bdd4bf5d4e48059d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.564975 9850 tablet_service.cc:1511] Processing CreateTablet for tablet 7c10b37fe2e54d7a951545147ac2e381 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260430 07:58:02.563133 10058 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:02.561759 9851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c37b1d9d0b54e12b8ce6cd28f1fdb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.574054 9850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c10b37fe2e54d7a951545147ac2e381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.575886 9847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a52542f8aa4aeca8e319d4f60e9699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.575868 9848 tablet_service.cc:1511] Processing CreateTablet for tablet 75807667b9164e0ba12c5013baff3b81 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260430 07:58:02.582659 9998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5d1b9983db48e198134c6ab5246fbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.596865 9848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75807667b9164e0ba12c5013baff3b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.609254 9924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de66bf2ed62e49aa95b1deff0383d037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.609838 9997 tablet_service.cc:1511] Processing CreateTablet for tablet de66bf2ed62e49aa95b1deff0383d037 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260430 07:58:02.611135 9927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b978c5a9ab41402c806352f5ace14295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.611828 9921 tablet_service.cc:1511] Processing CreateTablet for tablet 18a52542f8aa4aeca8e319d4f60e9699 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260430 07:58:02.611967 9996 tablet_service.cc:1511] Processing CreateTablet for tablet 9c37b1d9d0b54e12b8ce6cd28f1fdb07 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260430 07:58:02.613090 9921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a52542f8aa4aeca8e319d4f60e9699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.611109 9997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de66bf2ed62e49aa95b1deff0383d037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.625094 9996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c37b1d9d0b54e12b8ce6cd28f1fdb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.633718 9920 tablet_service.cc:1511] Processing CreateTablet for tablet 9c37b1d9d0b54e12b8ce6cd28f1fdb07 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260430 07:58:02.637352 10052 tablet_bootstrap.cc:492] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:02.645495 9853 tablet_service.cc:1511] Processing CreateTablet for tablet 02a63423ae464eb7b415a38419524150 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260430 07:58:02.648056 9920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c37b1d9d0b54e12b8ce6cd28f1fdb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.653447 10001 tablet_service.cc:1511] Processing CreateTablet for tablet 7c10b37fe2e54d7a951545147ac2e381 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260430 07:58:02.654194 9853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02a63423ae464eb7b415a38419524150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.654938 10001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c10b37fe2e54d7a951545147ac2e381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.662860 10054 tablet_bootstrap.cc:492] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:02.671445 10061 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:02.673938 10061 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.676657 10061 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:02.715502 9995 tablet_service.cc:1511] Processing CreateTablet for tablet 02a63423ae464eb7b415a38419524150 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260430 07:58:02.716992 9995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02a63423ae464eb7b415a38419524150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.731789 10002 tablet_service.cc:1511] Processing CreateTablet for tablet 75807667b9164e0ba12c5013baff3b81 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260430 07:58:02.733345 10002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75807667b9164e0ba12c5013baff3b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.738968 9849 tablet_service.cc:1511] Processing CreateTablet for tablet d2df038061ff495dbdb2eaa4ad73c76e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260430 07:58:02.740355 9849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2df038061ff495dbdb2eaa4ad73c76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.741513 10053 tablet_bootstrap.cc:492] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:02.742058 10053 ts_tablet_manager.cc:1403] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.267s user 0.019s sys 0.010s
I20260430 07:58:02.750293 9928 tablet_service.cc:1511] Processing CreateTablet for tablet 75807667b9164e0ba12c5013baff3b81 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260430 07:58:02.751756 9928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75807667b9164e0ba12c5013baff3b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.767802 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:02.768836 10054 tablet_bootstrap.cc:654] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:02.770920 9926 tablet_service.cc:1511] Processing CreateTablet for tablet 02a63423ae464eb7b415a38419524150 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260430 07:58:02.772358 9926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02a63423ae464eb7b415a38419524150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.745172 10053 raft_consensus.cc:359] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.780217 9844 tablet_service.cc:1511] Processing CreateTablet for tablet cb149eebc54e462c951d45bca72b76e8 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260430 07:58:02.786834 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:02.725031 9922 tablet_service.cc:1511] Processing CreateTablet for tablet 7c10b37fe2e54d7a951545147ac2e381 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
W20260430 07:58:02.791168 9978 leader_election.cc:343] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:02.793465 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb149eebc54e462c951d45bca72b76e8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260430 07:58:02.793632 9904 leader_election.cc:343] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:02.794338 9925 tablet_service.cc:1511] Processing CreateTablet for tablet cb149eebc54e462c951d45bca72b76e8 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260430 07:58:02.795646 9925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb149eebc54e462c951d45bca72b76e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.771226 9927 tablet_service.cc:1511] Processing CreateTablet for tablet d2df038061ff495dbdb2eaa4ad73c76e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260430 07:58:02.793512 10000 tablet_service.cc:1511] Processing CreateTablet for tablet cb149eebc54e462c951d45bca72b76e8 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260430 07:58:02.798619 9927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2df038061ff495dbdb2eaa4ad73c76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.798840 10052 tablet_bootstrap.cc:654] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:02.799037 10000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb149eebc54e462c951d45bca72b76e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.806499 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:02.808128 9903 leader_election.cc:343] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:02.808626 9903 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:02.809986 10060 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:02.811532 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:02.812424 10001 tablet_service.cc:1511] Processing CreateTablet for tablet d2df038061ff495dbdb2eaa4ad73c76e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
W20260430 07:58:02.813107 9828 leader_election.cc:343] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:02.813954 10001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2df038061ff495dbdb2eaa4ad73c76e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.823103 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:02.823556 9999 tablet_service.cc:1511] Processing CreateTablet for tablet cd7ed510277642ca8409dce1775947c1 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260430 07:58:02.825317 9999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd7ed510277642ca8409dce1775947c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260430 07:58:02.828899 9977 leader_election.cc:343] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:02.829442 9977 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:02.830657 10061 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:02.788774 10053 raft_consensus.cc:385] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:02.838109 9922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c10b37fe2e54d7a951545147ac2e381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.843212 9993 tablet_service.cc:1511] Processing CreateTablet for tablet 725a25b9eaa049c3ba8535dbe1100561 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260430 07:58:02.844601 9993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 725a25b9eaa049c3ba8535dbe1100561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.848127 9923 tablet_service.cc:1511] Processing CreateTablet for tablet 2ad545d8bf83435f87a2b1169a24878e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260430 07:58:02.854194 9851 tablet_service.cc:1511] Processing CreateTablet for tablet 2ad545d8bf83435f87a2b1169a24878e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260430 07:58:02.855690 9851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad545d8bf83435f87a2b1169a24878e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.856256 9996 tablet_service.cc:1511] Processing CreateTablet for tablet 202cb0ccf6a947e6bdcffc383288322c (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260430 07:58:02.852475 10053 raft_consensus.cc:740] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:02.857659 9996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 202cb0ccf6a947e6bdcffc383288322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.851483 9849 tablet_service.cc:1511] Processing CreateTablet for tablet cd7ed510277642ca8409dce1775947c1 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260430 07:58:02.860715 9846 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37818 (request call id 30) took 1076 ms. Trace:
I20260430 07:58:02.861120 9995 tablet_service.cc:1511] Processing CreateTablet for tablet 2ad545d8bf83435f87a2b1169a24878e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260430 07:58:02.861259 9846 rpcz_store.cc:276] 0430 07:58:01.784623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.446238 (+661615us) service_pool.cc:225] Handling call
0430 07:58:02.447893 (+ 1655us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.447987 (+ 94us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.631966 (+183979us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:02.764530 (+132564us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:02.860658 (+ 96128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:02.862407 9995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad545d8bf83435f87a2b1169a24878e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.862560 9997 tablet_service.cc:1511] Processing CreateTablet for tablet d3263a2b092d47c3b86f752e2bafcb63 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260430 07:58:02.860659 9847 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37818 (request call id 32) took 1073 ms. Trace:
I20260430 07:58:02.864226 9847 rpcz_store.cc:276] 0430 07:58:01.786770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.542172 (+755402us) service_pool.cc:225] Handling call
0430 07:58:02.560000 (+ 17828us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.568968 (+ 8968us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.726648 (+157680us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:02.859746 (+133098us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:02.860560 (+ 814us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5088}
I20260430 07:58:02.860658 10053 consensus_queue.cc:260] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:02.866704 9846 tablet_service.cc:1511] Processing CreateTablet for tablet d3263a2b092d47c3b86f752e2bafcb63 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260430 07:58:02.870934 9994 tablet_service.cc:1511] Processing CreateTablet for tablet 38bcd78c11574f1a9e016f9bb9365ca5 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260430 07:58:02.895709 9997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3263a2b092d47c3b86f752e2bafcb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.904300 10053 ts_tablet_manager.cc:1434] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.162s user 0.006s sys 0.006s
I20260430 07:58:02.905651 9994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38bcd78c11574f1a9e016f9bb9365ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.861114 9849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd7ed510277642ca8409dce1775947c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.878057 9919 tablet_service.cc:1511] Processing CreateTablet for tablet cd7ed510277642ca8409dce1775947c1 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260430 07:58:02.881222 9923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ad545d8bf83435f87a2b1169a24878e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.909534 9919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd7ed510277642ca8409dce1775947c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.881587 9847 tablet_service.cc:1511] Processing CreateTablet for tablet 725a25b9eaa049c3ba8535dbe1100561 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260430 07:58:02.882134 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:02.911912 9828 leader_election.cc:343] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:02.912380 9828 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:02.912551 9847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 725a25b9eaa049c3ba8535dbe1100561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.913492 10058 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:02.890031 9845 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37818 (request call id 27) took 1173 ms. Trace:
I20260430 07:58:02.919488 9845 rpcz_store.cc:276] 0430 07:58:01.716252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.414565 (+698313us) service_pool.cc:225] Handling call
0430 07:58:02.492369 (+ 77804us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.500214 (+ 7845us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.760035 (+259821us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:02.888449 (+128414us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:02.889720 (+ 1271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:02.922897 9845 tablet_service.cc:1511] Processing CreateTablet for tablet 202cb0ccf6a947e6bdcffc383288322c (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260430 07:58:02.942142 10053 tablet_bootstrap.cc:492] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
W20260430 07:58:02.962025 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 50155.8us
I20260430 07:58:02.963564 10052 tablet_bootstrap.cc:492] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:02.967346 9848 tablet_service.cc:1511] Processing CreateTablet for tablet 38bcd78c11574f1a9e016f9bb9365ca5 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260430 07:58:02.968652 9852 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37818 (request call id 28) took 1185 ms. Trace:
I20260430 07:58:02.969163 9852 rpcz_store.cc:276] 0430 07:58:01.783512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.422734 (+639222us) service_pool.cc:225] Handling call
0430 07:58:02.454508 (+ 31774us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.454609 (+ 101us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.838393 (+383784us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:02.961146 (+122753us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:02.968570 (+ 7424us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:02.971344 9852 tablet_service.cc:1511] Processing CreateTablet for tablet 3028e67ff7d948ee8accee91fbcb70c3 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260430 07:58:02.976410 10053 tablet_bootstrap.cc:654] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
W20260430 07:58:02.964393 9748 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 59590us
W20260430 07:58:02.981953 9964 tablet.cc:2404] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260430 07:58:02.986019 10052 ts_tablet_manager.cc:1403] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.380s user 0.015s sys 0.012s
I20260430 07:58:02.965896 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 202cb0ccf6a947e6bdcffc383288322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.991591 9998 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44006 (request call id 27) took 1012 ms. Trace:
I20260430 07:58:02.992951 9848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38bcd78c11574f1a9e016f9bb9365ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.995417 9921 tablet_service.cc:1511] Processing CreateTablet for tablet d3263a2b092d47c3b86f752e2bafcb63 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260430 07:58:02.996589 10001 tablet_service.cc:1511] Processing CreateTablet for tablet 829ed212483c445ba0dd4ae328bde3c2 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260430 07:58:02.992029 9998 rpcz_store.cc:276] 0430 07:58:01.979496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.425120 (+445624us) service_pool.cc:225] Handling call
0430 07:58:02.582088 (+156968us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.582207 (+ 119us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.821134 (+238927us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:02.991025 (+169891us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:02.991524 (+ 499us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:02.998073 9852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3028e67ff7d948ee8accee91fbcb70c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.001541 9846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3263a2b092d47c3b86f752e2bafcb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:02.969111 10002 tablet_service.cc:1511] Processing CreateTablet for tablet 3028e67ff7d948ee8accee91fbcb70c3 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
W20260430 07:58:02.981554 9809 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:58:03.004571 10052 raft_consensus.cc:359] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.005524 10052 raft_consensus.cc:385] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.005883 10052 raft_consensus.cc:740] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.006518 9850 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37818 (request call id 34) took 1071 ms. Trace:
I20260430 07:58:03.006762 9998 tablet_service.cc:1511] Processing CreateTablet for tablet a7e202af141f43f7b88e4231eed0ba94 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260430 07:58:03.007012 9850 rpcz_store.cc:276] 0430 07:58:01.934738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.564053 (+629315us) service_pool.cc:225] Handling call
0430 07:58:02.569457 (+ 5404us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.569570 (+ 113us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.851678 (+282108us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.005908 (+154230us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.006452 (+ 544us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:03.014665 9850 tablet_service.cc:1511] Processing CreateTablet for tablet 829ed212483c445ba0dd4ae328bde3c2 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260430 07:58:03.011634 10000 tablet_service.cc:1511] Processing CreateTablet for tablet 2e8dad8acea649479d1e927cd32599fe (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260430 07:58:03.018998 9922 tablet_service.cc:1511] Processing CreateTablet for tablet 725a25b9eaa049c3ba8535dbe1100561 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260430 07:58:02.998107 10001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829ed212483c445ba0dd4ae328bde3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.007061 10052 consensus_queue.cc:260] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.035256 9853 tablet_service.cc:1511] Processing CreateTablet for tablet 2e8dad8acea649479d1e927cd32599fe (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260430 07:58:02.997061 9921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3263a2b092d47c3b86f752e2bafcb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.045954 9853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e8dad8acea649479d1e927cd32599fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.046983 10054 tablet_bootstrap.cc:492] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:03.047498 10054 ts_tablet_manager.cc:1403] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.408s user 0.010s sys 0.009s
I20260430 07:58:03.047974 9927 tablet_service.cc:1511] Processing CreateTablet for tablet 202cb0ccf6a947e6bdcffc383288322c (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260430 07:58:03.029049 9851 tablet_service.cc:1511] Processing CreateTablet for tablet a7e202af141f43f7b88e4231eed0ba94 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260430 07:58:03.050436 9851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e202af141f43f7b88e4231eed0ba94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.050515 9920 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 33) took 1026 ms. Trace:
I20260430 07:58:03.050853 10054 raft_consensus.cc:359] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.050923 9920 rpcz_store.cc:276] 0430 07:58:02.024570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.632732 (+608162us) service_pool.cc:225] Handling call
0430 07:58:02.634729 (+ 1997us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.634830 (+ 101us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.898191 (+263361us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.049992 (+151801us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.050455 (+ 463us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:03.051755 10054 raft_consensus.cc:385] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.052508 10054 raft_consensus.cc:740] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.053619 10054 consensus_queue.cc:260] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.059712 9924 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 31) took 1037 ms. Trace:
I20260430 07:58:03.060176 9924 rpcz_store.cc:276] 0430 07:58:02.022387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.480614 (+458227us) service_pool.cc:225] Handling call
0430 07:58:02.482311 (+ 1697us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.482521 (+ 210us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.978632 (+496111us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.059173 (+ 80541us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.059646 (+ 473us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:03.061960 10054 ts_tablet_manager.cc:1434] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260430 07:58:03.065773 10002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3028e67ff7d948ee8accee91fbcb70c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.069082 9848 tablet_service.cc:1511] Processing CreateTablet for tablet 19c3ca044c3d4377b31c4d07c513415c (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260430 07:58:03.070070 9850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829ed212483c445ba0dd4ae328bde3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.070200 9998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e202af141f43f7b88e4231eed0ba94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.074354 9928 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 35) took 1014 ms. Trace:
I20260430 07:58:03.074589 10052 ts_tablet_manager.cc:1434] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.088s user 0.003s sys 0.003s
I20260430 07:58:03.075708 9923 tablet_service.cc:1511] Processing CreateTablet for tablet 3028e67ff7d948ee8accee91fbcb70c3 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260430 07:58:03.075147 9847 tablet_service.cc:1511] Processing CreateTablet for tablet 2947a88351384118b009f71f871ab9c8 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260430 07:58:03.078047 9847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947a88351384118b009f71f871ab9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.079810 9844 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37818 (request call id 38) took 1003 ms. Trace:
I20260430 07:58:03.080457 9927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 202cb0ccf6a947e6bdcffc383288322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.085343 10053 tablet_bootstrap.cc:492] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:03.085867 10053 ts_tablet_manager.cc:1403] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.144s user 0.009s sys 0.003s
I20260430 07:58:03.086512 9849 tablet_service.cc:1511] Processing CreateTablet for tablet 3d83257ff86a443fbf15a8130dd8106e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260430 07:58:03.101842 9849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d83257ff86a443fbf15a8130dd8106e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.102850 9926 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 36) took 1036 ms. Trace:
I20260430 07:58:03.087853 10000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e8dad8acea649479d1e927cd32599fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.103294 9926 rpcz_store.cc:276] 0430 07:58:02.066010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.754077 (+688067us) service_pool.cc:225] Handling call
0430 07:58:02.771829 (+ 17752us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.771938 (+ 109us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.883287 (+111349us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.075642 (+192355us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.102791 (+ 27149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5716,"spinlock_wait_cycles":1289600}
I20260430 07:58:03.107473 9919 tablet_service.cc:1511] Processing CreateTablet for tablet 829ed212483c445ba0dd4ae328bde3c2 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260430 07:58:03.107782 9995 tablet_service.cc:1511] Processing CreateTablet for tablet 82b70cfe134341abb8df72ad2244032e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260430 07:58:03.053169 9920 tablet_service.cc:1511] Processing CreateTablet for tablet 38bcd78c11574f1a9e016f9bb9365ca5 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260430 07:58:03.104197 9925 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 38) took 1017 ms. Trace:
I20260430 07:58:03.109738 9926 tablet_service.cc:1511] Processing CreateTablet for tablet a7e202af141f43f7b88e4231eed0ba94 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260430 07:58:03.111474 9846 tablet_service.cc:1511] Processing CreateTablet for tablet df6bdf20bc534fb2a8de1b1c2cbd9af9 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260430 07:58:03.090207 10054 tablet_bootstrap.cc:492] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:03.089010 10053 raft_consensus.cc:359] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.126418 10053 raft_consensus.cc:385] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.126787 10053 raft_consensus.cc:740] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.127540 10053 consensus_queue.cc:260] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.091234 9999 tablet_service.cc:1511] Processing CreateTablet for tablet 2947a88351384118b009f71f871ab9c8 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260430 07:58:03.074824 9928 rpcz_store.cc:276] 0430 07:58:02.060133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.726827 (+666694us) service_pool.cc:225] Handling call
0430 07:58:02.751179 (+ 24352us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.751272 (+ 93us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.987633 (+236361us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.073762 (+ 86129us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.074284 (+ 522us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:03.130630 9928 tablet_service.cc:1511] Processing CreateTablet for tablet 2947a88351384118b009f71f871ab9c8 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260430 07:58:03.134945 10053 ts_tablet_manager.cc:1434] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.049s user 0.007s sys 0.000s
I20260430 07:58:03.136126 10053 tablet_bootstrap.cc:492] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:03.137003 9994 tablet_service.cc:1511] Processing CreateTablet for tablet 9031ab0cc4144643b99f584e93b8c75f (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260430 07:58:03.093335 9845 tablet_service.cc:1511] Processing CreateTablet for tablet 82b70cfe134341abb8df72ad2244032e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260430 07:58:03.093688 9993 tablet_service.cc:1511] Processing CreateTablet for tablet 19c3ca044c3d4377b31c4d07c513415c (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260430 07:58:03.095857 9852 tablet_service.cc:1511] Processing CreateTablet for tablet 9031ab0cc4144643b99f584e93b8c75f (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260430 07:58:03.110121 9925 rpcz_store.cc:276] 0430 07:58:02.087320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.787752 (+700432us) service_pool.cc:225] Handling call
0430 07:58:02.795165 (+ 7413us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.795272 (+ 107us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.828691 (+ 33419us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.103552 (+274861us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.104137 (+ 585us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:03.150301 9847 tablet_service.cc:1511] Processing CreateTablet for tablet 95ce138277864f41ba2e21b8b8d4a730 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260430 07:58:03.150897 9925 tablet_service.cc:1511] Processing CreateTablet for tablet 19c3ca044c3d4377b31c4d07c513415c (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260430 07:58:03.112659 9997 tablet_service.cc:1511] Processing CreateTablet for tablet 3d83257ff86a443fbf15a8130dd8106e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260430 07:58:03.082732 9844 rpcz_store.cc:276] 0430 07:58:02.076133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:02.779179 (+703046us) service_pool.cc:225] Handling call
0430 07:58:02.791672 (+ 12493us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:02.793028 (+ 1356us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:02.968538 (+175510us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.079322 (+110784us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.079748 (+ 426us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:03.156874 9998 tablet_service.cc:1511] Processing CreateTablet for tablet 805814cfaf2b43ab87c90b2b7b56a211 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260430 07:58:03.158119 9844 tablet_service.cc:1511] Processing CreateTablet for tablet 805814cfaf2b43ab87c90b2b7b56a211 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260430 07:58:03.114183 9848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c3ca044c3d4377b31c4d07c513415c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.114071 9996 tablet_service.cc:1511] Processing CreateTablet for tablet df6bdf20bc534fb2a8de1b1c2cbd9af9 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260430 07:58:03.161885 9847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ce138277864f41ba2e21b8b8d4a730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.114946 10052 tablet_bootstrap.cc:492] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:03.089830 9923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3028e67ff7d948ee8accee91fbcb70c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.169945 9920 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38bcd78c11574f1a9e016f9bb9365ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.118800 9924 tablet_service.cc:1511] Processing CreateTablet for tablet 2e8dad8acea649479d1e927cd32599fe (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260430 07:58:03.109108 9995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b70cfe134341abb8df72ad2244032e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.171221 9924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e8dad8acea649479d1e927cd32599fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.172498 9849 tablet_service.cc:1511] Processing CreateTablet for tablet 00df97e04911453d95498aacc19c7685 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260430 07:58:03.173375 9850 tablet_service.cc:1511] Processing CreateTablet for tablet 058d2a87aec64ba1b745c5bd593c00e2 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260430 07:58:03.173909 9849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00df97e04911453d95498aacc19c7685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.175063 9850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 058d2a87aec64ba1b745c5bd593c00e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.177814 9997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d83257ff86a443fbf15a8130dd8106e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.180460 9996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6bdf20bc534fb2a8de1b1c2cbd9af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.181916 9919 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829ed212483c445ba0dd4ae328bde3c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.184741 9925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c3ca044c3d4377b31c4d07c513415c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.185067 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b70cfe134341abb8df72ad2244032e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.185067 9994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9031ab0cc4144643b99f584e93b8c75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.188450 9846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6bdf20bc534fb2a8de1b1c2cbd9af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.189872 9922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 725a25b9eaa049c3ba8535dbe1100561. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.193425 9993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19c3ca044c3d4377b31c4d07c513415c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.193357 9852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9031ab0cc4144643b99f584e93b8c75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.193661 9928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947a88351384118b009f71f871ab9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.197224 9999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947a88351384118b009f71f871ab9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.201766 9926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e202af141f43f7b88e4231eed0ba94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.203143 10052 tablet_bootstrap.cc:654] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.205931 9921 tablet_service.cc:1511] Processing CreateTablet for tablet 3d83257ff86a443fbf15a8130dd8106e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260430 07:58:03.209481 9927 tablet_service.cc:1511] Processing CreateTablet for tablet 95ce138277864f41ba2e21b8b8d4a730 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260430 07:58:03.211732 10000 tablet_service.cc:1511] Processing CreateTablet for tablet 00df97e04911453d95498aacc19c7685 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260430 07:58:03.213091 10000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00df97e04911453d95498aacc19c7685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.214654 9921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d83257ff86a443fbf15a8130dd8106e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.219427 9927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ce138277864f41ba2e21b8b8d4a730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.222384 10054 tablet_bootstrap.cc:654] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.229894 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 805814cfaf2b43ab87c90b2b7b56a211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.232777 9998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 805814cfaf2b43ab87c90b2b7b56a211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.237413 9995 tablet_service.cc:1511] Processing CreateTablet for tablet 95ce138277864f41ba2e21b8b8d4a730 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260430 07:58:03.245934 10002 tablet_service.cc:1511] Processing CreateTablet for tablet 058d2a87aec64ba1b745c5bd593c00e2 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260430 07:58:03.247396 10002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 058d2a87aec64ba1b745c5bd593c00e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.256677 9995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ce138277864f41ba2e21b8b8d4a730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.263984 10060 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.264736 10060 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.289815 10061 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.290398 10061 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.291844 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:03.292603 10061 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:03.293557 9903 leader_election.cc:343] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.294528 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:03.295380 9863 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:03.294068 10054 tablet_bootstrap.cc:492] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:03.296415 10054 ts_tablet_manager.cc:1403] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.206s user 0.016s sys 0.011s
I20260430 07:58:03.296707 9978 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:03.298182 10061 raft_consensus.cc:2804] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:03.298645 10061 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:03.298995 10061 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.299737 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:03.302832 9977 leader_election.cc:343] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.303098 10053 tablet_bootstrap.cc:654] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.303973 10060 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.305576 10054 raft_consensus.cc:359] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.306354 10054 raft_consensus.cc:385] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.306695 10054 raft_consensus.cc:740] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.307425 10054 consensus_queue.cc:260] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.309604 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:03.311149 9904 leader_election.cc:343] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.311687 9904 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:03.314663 10060 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:03.326781 10061 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.333247 10052 tablet_bootstrap.cc:492] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:03.333859 10052 ts_tablet_manager.cc:1403] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.219s user 0.012s sys 0.007s
I20260430 07:58:03.336805 10052 raft_consensus.cc:359] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.337831 10052 raft_consensus.cc:385] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.338150 10052 raft_consensus.cc:740] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.351145 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:03.351857 9863 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.353256 10054 ts_tablet_manager.cc:1434] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.057s user 0.034s sys 0.016s
I20260430 07:58:03.354374 10054 tablet_bootstrap.cc:492] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:03.360260 9863 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:03.360785 10054 tablet_bootstrap.cc:654] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.361500 9978 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:03.371452 10061 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:03.371459 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:03.372203 10061 raft_consensus.cc:2804] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:03.372675 10061 raft_consensus.cc:697] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:03.373015 10054 tablet_bootstrap.cc:492] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
W20260430 07:58:03.373128 9977 leader_election.cc:343] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.373605 10054 ts_tablet_manager.cc:1403] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.011s
I20260430 07:58:03.373540 10061 consensus_queue.cc:237] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.377519 10054 raft_consensus.cc:359] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.383188 9927 tablet_service.cc:1511] Processing CreateTablet for tablet df6bdf20bc534fb2a8de1b1c2cbd9af9 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260430 07:58:03.383863 9921 tablet_service.cc:1511] Processing CreateTablet for tablet 9031ab0cc4144643b99f584e93b8c75f (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260430 07:58:03.378273 10054 raft_consensus.cc:385] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.387043 10054 raft_consensus.cc:740] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.387845 10054 consensus_queue.cc:260] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.387866 10052 consensus_queue.cc:260] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.390288 10054 ts_tablet_manager.cc:1434] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.016s user 0.007s sys 0.001s
I20260430 07:58:03.391317 10054 tablet_bootstrap.cc:492] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:03.397109 9921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9031ab0cc4144643b99f584e93b8c75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.401678 9927 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6bdf20bc534fb2a8de1b1c2cbd9af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.415436 9923 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 45) took 1081 ms. Trace:
I20260430 07:58:03.416222 10053 tablet_bootstrap.cc:492] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:03.416396 10082 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.416731 10053 ts_tablet_manager.cc:1403] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.281s user 0.016s sys 0.002s
I20260430 07:58:03.416888 10082 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.417016 9925 tablet_service.cc:1511] Processing CreateTablet for tablet 058d2a87aec64ba1b745c5bd593c00e2 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260430 07:58:03.418340 9925 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 058d2a87aec64ba1b745c5bd593c00e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.415856 9923 rpcz_store.cc:276] 0430 07:58:02.333653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:03.059610 (+725957us) service_pool.cc:225] Handling call
0430 07:58:03.076992 (+ 17382us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:03.077088 (+ 96us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:03.227301 (+150213us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.414910 (+187609us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.415359 (+ 449us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9792}
I20260430 07:58:03.419728 9924 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 48) took 1017 ms. Trace:
I20260430 07:58:03.420506 10054 tablet_bootstrap.cc:654] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.420715 9926 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 47) took 1032 ms. Trace:
I20260430 07:58:03.421118 9923 tablet_service.cc:1511] Processing CreateTablet for tablet 82b70cfe134341abb8df72ad2244032e (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260430 07:58:03.422952 10053 raft_consensus.cc:359] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.421101 9924 rpcz_store.cc:276] 0430 07:58:02.402657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:03.117490 (+714833us) service_pool.cc:225] Handling call
0430 07:58:03.170654 (+ 53164us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:03.170848 (+ 194us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:03.230147 (+ 59299us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.400413 (+170266us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.419666 (+ 19253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26977}
I20260430 07:58:03.432981 9923 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82b70cfe134341abb8df72ad2244032e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.428694 9761 catalog_manager.cc:5671] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:03.429401 9926 rpcz_store.cc:276] 0430 07:58:02.388042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:03.108799 (+720757us) service_pool.cc:225] Handling call
0430 07:58:03.110551 (+ 1752us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:03.110644 (+ 93us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:03.316635 (+205991us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.420188 (+103553us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.420660 (+ 472us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18579}
I20260430 07:58:03.430593 9922 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 42) took 1138 ms. Trace:
I20260430 07:58:03.431782 9928 tablet_service.cc:1511] Processing CreateTablet for tablet 805814cfaf2b43ab87c90b2b7b56a211 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260430 07:58:03.440066 10053 raft_consensus.cc:385] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.438810 10052 ts_tablet_manager.cc:1434] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.105s user 0.007s sys 0.001s
I20260430 07:58:03.440285 10058 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.446242 10082 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:03.443343 10058 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.448663 10058 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.449725 10060 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.450528 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:03.450412 9924 tablet_service.cc:1511] Processing CreateTablet for tablet 00df97e04911453d95498aacc19c7685 (DEFAULT_TABLE table=test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260430 07:58:03.450359 10060 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:03.452210 9828 leader_election.cc:343] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.453512 10060 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.454602 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:03.455066 9928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 805814cfaf2b43ab87c90b2b7b56a211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:03.455271 9863 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:03.455427 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:03.456791 9904 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:03.457085 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:03.457695 9937 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:03.458249 10060 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260430 07:58:03.459153 9977 leader_election.cc:343] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.459221 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:03.459944 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:03.462436 10058 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260430 07:58:03.462739 9903 leader_election.cc:343] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.462930 10058 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:03.463380 10058 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.439399 9922 rpcz_store.cc:276] 0430 07:58:02.292344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:03.017516 (+725172us) service_pool.cc:225] Handling call
0430 07:58:03.082924 (+ 65408us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:03.102351 (+ 19427us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:03.326296 (+223945us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.430114 (+103818us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.430540 (+ 426us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13059}
I20260430 07:58:03.464550 10060 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:03.465801 10060 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.460747 9924 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00df97e04911453d95498aacc19c7685. 1 dirs total, 0 dirs full, 0 dirs failed
W20260430 07:58:03.426437 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 60380.5us
I20260430 07:58:03.471591 9920 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 44) took 1147 ms. Trace:
I20260430 07:58:03.472007 9920 rpcz_store.cc:276] 0430 07:58:02.324350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:03.052211 (+727861us) service_pool.cc:225] Handling call
0430 07:58:03.109965 (+ 57754us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:03.110056 (+ 91us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:03.342825 (+232769us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.471047 (+128222us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.471512 (+ 465us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:03.443022 10053 raft_consensus.cc:740] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.474282 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:03.474921 10053 consensus_queue.cc:260] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:03.478207 9978 leader_election.cc:343] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.478991 9978 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:03.480041 10082 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:03.487604 10058 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.490657 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:03.494086 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:03.494935 10052 tablet_bootstrap.cc:492] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
W20260430 07:58:03.495690 9828 leader_election.cc:343] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.496415 10058 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.503279 10060 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.504614 9937 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 1: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:03.507474 9919 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35252 (request call id 46) took 1166 ms. Trace:
I20260430 07:58:03.507843 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:03.507915 9919 rpcz_store.cc:276] 0430 07:58:02.340647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:03.106337 (+765690us) service_pool.cc:225] Handling call
0430 07:58:03.108337 (+ 2000us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0430 07:58:03.108452 (+ 115us) ts_tablet_manager.cc:682] Creating new metadata...
0430 07:58:03.432346 (+323894us) tablet_metadata.cc:665] Metadata flushed
0430 07:58:03.506862 (+ 74516us) tablet_replica.cc:206] Creating consensus instance
0430 07:58:03.507413 (+ 551us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:03.509660 10058 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260430 07:58:03.512458 10060 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.513607 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:03.514966 9903 leader_election.cc:343] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.516105 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:03.517656 10054 tablet_bootstrap.cc:492] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:03.518303 10054 ts_tablet_manager.cc:1403] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.127s user 0.015s sys 0.000s
I20260430 07:58:03.518994 10052 tablet_bootstrap.cc:654] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.517714 9863 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 1: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:03.520462 9904 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:03.522121 10060 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260430 07:58:03.524034 10053 ts_tablet_manager.cc:1434] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.107s user 0.017s sys 0.004s
I20260430 07:58:03.525449 10054 raft_consensus.cc:359] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.526212 10053 tablet_bootstrap.cc:492] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:03.526177 10054 raft_consensus.cc:385] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.526847 10054 raft_consensus.cc:740] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.528062 10054 consensus_queue.cc:260] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.534078 10053 tablet_bootstrap.cc:654] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.535413 10054 ts_tablet_manager.cc:1434] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260430 07:58:03.537192 10054 tablet_bootstrap.cc:492] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:03.562376 10054 tablet_bootstrap.cc:654] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.566764 10053 tablet_bootstrap.cc:492] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:03.567322 10053 ts_tablet_manager.cc:1403] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.041s user 0.021s sys 0.009s
I20260430 07:58:03.569797 10052 tablet_bootstrap.cc:492] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:03.570303 10052 ts_tablet_manager.cc:1403] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.076s user 0.013s sys 0.014s
I20260430 07:58:03.570789 10053 raft_consensus.cc:359] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.571616 10053 raft_consensus.cc:385] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.571918 10053 raft_consensus.cc:740] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.572650 10053 consensus_queue.cc:260] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.573320 10052 raft_consensus.cc:359] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.575042 10052 raft_consensus.cc:385] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.575350 10052 raft_consensus.cc:740] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.575765 10053 ts_tablet_manager.cc:1434] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260430 07:58:03.576128 10052 consensus_queue.cc:260] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.576784 10053 tablet_bootstrap.cc:492] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:03.581919 10052 ts_tablet_manager.cc:1434] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20260430 07:58:03.583285 10052 tablet_bootstrap.cc:492] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:03.588644 10054 tablet_bootstrap.cc:492] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:03.589205 10054 ts_tablet_manager.cc:1403] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.000s
I20260430 07:58:03.593899 10053 tablet_bootstrap.cc:654] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.594000 10087 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.594470 10054 raft_consensus.cc:359] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.595714 10054 raft_consensus.cc:385] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.595697 10087 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.596372 10054 raft_consensus.cc:740] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.598150 10054 consensus_queue.cc:260] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.600657 10054 ts_tablet_manager.cc:1434] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260430 07:58:03.601085 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:03.602742 10054 tablet_bootstrap.cc:492] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
W20260430 07:58:03.604789 9904 leader_election.cc:343] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.610831 10087 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.613986 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:03.617719 9903 leader_election.cc:343] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.618229 9903 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:03.620805 10087 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:03.627379 10054 tablet_bootstrap.cc:654] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.627880 10052 tablet_bootstrap.cc:654] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.629007 10053 tablet_bootstrap.cc:492] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:03.629765 10053 ts_tablet_manager.cc:1403] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.053s user 0.027s sys 0.023s
I20260430 07:58:03.633703 10053 raft_consensus.cc:359] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.634553 10053 raft_consensus.cc:385] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.634984 10053 raft_consensus.cc:740] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.635491 10052 tablet_bootstrap.cc:492] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:03.639124 10052 ts_tablet_manager.cc:1403] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.011s
I20260430 07:58:03.635717 10053 consensus_queue.cc:260] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.642442 10053 ts_tablet_manager.cc:1434] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260430 07:58:03.643404 10054 tablet_bootstrap.cc:492] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:03.643705 10053 tablet_bootstrap.cc:492] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:03.644348 10054 ts_tablet_manager.cc:1403] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.007s
I20260430 07:58:03.645707 10052 raft_consensus.cc:359] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.646508 10052 raft_consensus.cc:385] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.646837 10052 raft_consensus.cc:740] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.647570 10052 consensus_queue.cc:260] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.651921 10053 tablet_bootstrap.cc:654] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.651885 10054 raft_consensus.cc:359] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.652972 10054 raft_consensus.cc:385] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.653401 10054 raft_consensus.cc:740] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.654328 10054 consensus_queue.cc:260] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.656591 10082 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.658593 10082 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.662437 10052 ts_tablet_manager.cc:1434] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.023s user 0.005s sys 0.007s
I20260430 07:58:03.665647 10054 ts_tablet_manager.cc:1434] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.021s user 0.009s sys 0.004s
I20260430 07:58:03.667091 10053 tablet_bootstrap.cc:492] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:03.667147 10054 tablet_bootstrap.cc:492] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:03.667853 10053 ts_tablet_manager.cc:1403] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.004s
I20260430 07:58:03.671125 10053 raft_consensus.cc:359] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.671957 10053 raft_consensus.cc:385] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.672272 10053 raft_consensus.cc:740] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.672979 10053 consensus_queue.cc:260] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.675859 10054 tablet_bootstrap.cc:654] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.675915 10053 ts_tablet_manager.cc:1434] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260430 07:58:03.677438 10053 tablet_bootstrap.cc:492] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:03.679549 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:03.680933 9863 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:03.681905 10087 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.682574 10087 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.683310 9978 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:03.684509 10061 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:03.684927 10061 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:03.685258 10061 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.685436 10087 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.686848 10053 tablet_bootstrap.cc:654] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.686976 10052 tablet_bootstrap.cc:492] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:03.688647 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:03.689337 10012 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:03.689378 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:03.690965 9903 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:03.691932 9904 leader_election.cc:343] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.693466 10087 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:03.693948 10087 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:03.694368 10087 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.695473 10052 tablet_bootstrap.cc:654] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.703666 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:03.704640 9937 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:03.708081 10087 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.708854 10052 tablet_bootstrap.cc:492] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:03.711052 10052 ts_tablet_manager.cc:1403] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.012s
I20260430 07:58:03.713166 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:03.713209 10082 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:03.713996 10052 raft_consensus.cc:359] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.714768 10052 raft_consensus.cc:385] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.715067 10052 raft_consensus.cc:740] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.715823 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:03.716365 10012 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.716131 10061 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.719002 10052 consensus_queue.cc:260] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.720044 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:03.720717 9862 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.721422 9863 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.726572 10012 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:03.728030 9903 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:03.729123 10060 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:03.729514 9862 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:03.730751 9978 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:03.721575 10052 ts_tablet_manager.cc:1434] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20260430 07:58:03.731940 10082 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:03.732479 10082 raft_consensus.cc:697] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:03.732811 10052 tablet_bootstrap.cc:492] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:03.733412 10082 consensus_queue.cc:237] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.723819 10054 tablet_bootstrap.cc:492] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:03.737798 10054 ts_tablet_manager.cc:1403] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.015s
I20260430 07:58:03.739704 9863 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:03.745886 10052 tablet_bootstrap.cc:654] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.748988 10054 raft_consensus.cc:359] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.750327 10054 raft_consensus.cc:385] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.750707 10054 raft_consensus.cc:740] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.751652 10054 consensus_queue.cc:260] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.754444 10061 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:03.749785 10087 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.756121 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:03.757685 9937 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.750460 10060 raft_consensus.cc:697] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:03.761390 10060 consensus_queue.cc:237] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.759114 10054 ts_tablet_manager.cc:1434] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260430 07:58:03.771139 10054 tablet_bootstrap.cc:492] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:03.775115 9937 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:03.780862 10053 tablet_bootstrap.cc:492] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:03.781756 10053 ts_tablet_manager.cc:1403] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.105s user 0.043s sys 0.014s
I20260430 07:58:03.785625 10053 raft_consensus.cc:359] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.786585 10053 raft_consensus.cc:385] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.786942 10053 raft_consensus.cc:740] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.790978 10061 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.791680 10061 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.794248 10052 tablet_bootstrap.cc:492] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:03.794780 10052 ts_tablet_manager.cc:1403] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.062s user 0.020s sys 0.034s
I20260430 07:58:03.795359 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:03.787894 10053 consensus_queue.cc:260] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.800745 10053 ts_tablet_manager.cc:1434] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.019s user 0.007s sys 0.001s
I20260430 07:58:03.800195 10052 raft_consensus.cc:359] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.801324 10052 raft_consensus.cc:385] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.801836 10052 raft_consensus.cc:740] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.802615 10061 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:03.803388 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:03.802790 10052 consensus_queue.cc:260] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.804024 9937 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:03.804188 9761 catalog_manager.cc:5671] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:03.805121 9977 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:03.805945 10061 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:03.806309 10061 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:03.806639 10061 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:03.808957 9978 leader_election.cc:343] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.811081 10054 tablet_bootstrap.cc:654] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.814042 10053 tablet_bootstrap.cc:492] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:03.816098 10061 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.818276 10052 ts_tablet_manager.cc:1434] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.023s user 0.011s sys 0.000s
I20260430 07:58:03.819972 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:03.823366 9978 leader_election.cc:343] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.819720 10052 tablet_bootstrap.cc:492] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:03.820709 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:03.824970 9937 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.826717 10053 tablet_bootstrap.cc:654] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.821132 10061 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:03.839409 9937 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:03.844305 10052 tablet_bootstrap.cc:654] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.844695 9977 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:03.844686 9761 catalog_manager.cc:5671] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:03.846107 10061 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:03.846587 10061 raft_consensus.cc:697] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:03.847644 10061 consensus_queue.cc:237] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.850060 10054 tablet_bootstrap.cc:492] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:03.850378 10052 tablet_bootstrap.cc:492] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:03.850548 10054 ts_tablet_manager.cc:1403] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.080s user 0.022s sys 0.004s
I20260430 07:58:03.850787 10052 ts_tablet_manager.cc:1403] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.008s
I20260430 07:58:03.853917 10054 raft_consensus.cc:359] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.854312 10082 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.854808 10054 raft_consensus.cc:385] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.855141 10054 raft_consensus.cc:740] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.855181 10082 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.855989 10054 consensus_queue.cc:260] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.858356 10082 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:03.859112 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:03.860929 9978 leader_election.cc:343] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.862716 10054 ts_tablet_manager.cc:1434] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20260430 07:58:03.863742 10054 tablet_bootstrap.cc:492] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:03.865950 10053 tablet_bootstrap.cc:492] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:03.866549 10053 ts_tablet_manager.cc:1403] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.053s user 0.018s sys 0.008s
I20260430 07:58:03.861128 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:03.868896 9977 leader_election.cc:343] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.868919 10052 raft_consensus.cc:359] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.869715 9977 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:03.870059 10060 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.870014 10052 raft_consensus.cc:385] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.870510 10052 raft_consensus.cc:740] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.870594 10060 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.871315 10082 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:03.871272 10052 consensus_queue.cc:260] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.872572 10060 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.874928 10053 raft_consensus.cc:359] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.875737 10053 raft_consensus.cc:385] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.876058 10053 raft_consensus.cc:740] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.877079 10052 ts_tablet_manager.cc:1434] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260430 07:58:03.876858 10053 consensus_queue.cc:260] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.877400 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:03.878618 10052 tablet_bootstrap.cc:492] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:03.876662 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:03.879442 10053 ts_tablet_manager.cc:1434] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260430 07:58:03.879844 9862 raft_consensus.cc:2393] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 1: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:03.881328 10082 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.881884 10082 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.885109 10052 tablet_bootstrap.cc:654] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.885198 10054 tablet_bootstrap.cc:654] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.886533 9903 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:03.884991 10080 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.889236 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:03.889230 10080 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:03.891281 9978 leader_election.cc:343] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.892882 10080 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:03.893026 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:03.893390 10053 tablet_bootstrap.cc:492] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:03.893786 9862 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:03.894111 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:03.895432 9978 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:03.896373 10080 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:03.896840 10080 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:03.897374 10080 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:03.899384 9977 leader_election.cc:343] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.900097 10053 tablet_bootstrap.cc:654] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:03.900012 9977 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:03.901110 10061 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:03.902161 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:03.902830 10082 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:03.905521 10060 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.923074 10058 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.923816 10058 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.925212 9937 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:03.909698 9761 catalog_manager.cc:5671] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:03.927820 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:03.930068 9828 leader_election.cc:343] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:03.945132 10058 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:03.945667 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 46417.1us
I20260430 07:58:03.947539 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:03.948478 10012 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:03.950245 9828 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:03.952714 10060 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:03.954785 10082 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:03.957196 10058 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:03.957603 10058 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:03.957851 10058 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:03.959906 10082 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:03.969980 10054 tablet_bootstrap.cc:492] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:03.970504 10054 ts_tablet_manager.cc:1403] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.010s
I20260430 07:58:03.974464 10054 raft_consensus.cc:359] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.975167 10060 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.975648 10054 raft_consensus.cc:385] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:03.979696 10053 tablet_bootstrap.cc:492] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:03.980482 10053 ts_tablet_manager.cc:1403] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.087s user 0.022s sys 0.027s
I20260430 07:58:03.976678 10054 raft_consensus.cc:740] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.981567 10080 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.987210 10087 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:03.987743 10087 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.988158 10058 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.989892 10087 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.990531 10080 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:03.993419 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:03.994045 10058 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:03.995731 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:03.996345 9863 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.000385 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:04.000949 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:04.001075 10012 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:04.002039 10058 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:04.002255 9904 leader_election.cc:343] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.002507 10058 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.003594 9903 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:04.005048 10058 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:04.006445 9809 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:58:04.007634 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:04.007932 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:04.008416 10012 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.010102 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:04.010728 9936 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:04.013962 9828 leader_election.cc:343] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.014966 10087 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:04.015348 10087 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:04.015688 10087 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.018711 9863 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:03.983651 10052 tablet_bootstrap.cc:492] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:03.975721 10060 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:03.985195 10053 raft_consensus.cc:359] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.025219 10052 ts_tablet_manager.cc:1403] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.147s user 0.024s sys 0.057s
I20260430 07:58:04.025130 10053 raft_consensus.cc:385] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.025754 10053 raft_consensus.cc:740] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.025813 9978 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:04.019966 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:04.026508 10053 consensus_queue.cc:260] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:04.027805 9828 leader_election.cc:343] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.027895 10052 raft_consensus.cc:359] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.028290 9828 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:04.028611 10052 raft_consensus.cc:385] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.028679 10053 ts_tablet_manager.cc:1434] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20260430 07:58:04.028921 10052 raft_consensus.cc:740] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:03.983289 10054 consensus_queue.cc:260] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.029142 9937 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.029481 10061 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:04.030151 10061 raft_consensus.cc:697] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:04.031296 10054 ts_tablet_manager.cc:1434] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.060s user 0.009s sys 0.000s
I20260430 07:58:04.030997 10061 consensus_queue.cc:237] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.033192 9936 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:04.036088 10052 consensus_queue.cc:260] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.039429 10052 ts_tablet_manager.cc:1434] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260430 07:58:04.044914 10113 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:04.047035 10053 tablet_bootstrap.cc:492] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:04.048179 10052 tablet_bootstrap.cc:492] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:04.048659 10054 tablet_bootstrap.cc:492] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:04.050954 10087 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.058710 10087 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.059204 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:04.059579 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:04.059893 10011 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:04.060141 9863 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.064607 9903 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:04.066182 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:04.066844 10011 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.069926 10052 tablet_bootstrap.cc:654] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.071094 9863 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:04.072306 9904 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:04.073243 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:04.079008 9759 catalog_manager.cc:5671] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:04.080305 10054 tablet_bootstrap.cc:654] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.082862 10101 raft_consensus.cc:2804] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:04.083314 10101 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:04.018735 10012 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:04.083760 10101 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.083930 10052 tablet_bootstrap.cc:492] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:04.085144 9828 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:04.086709 10058 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:04.087203 10058 raft_consensus.cc:697] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:04.088294 10058 consensus_queue.cc:237] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.090296 10087 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:04.090842 10087 raft_consensus.cc:697] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:04.091077 9937 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:04.091768 10087 consensus_queue.cc:237] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.083364 10060 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.093583 10052 ts_tablet_manager.cc:1403] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.046s user 0.016s sys 0.005s
W20260430 07:58:04.093720 9904 leader_election.cc:343] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.094076 10011 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:04.111425 10053 tablet_bootstrap.cc:654] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.117794 10101 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.120728 10101 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.123418 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:04.123786 10052 raft_consensus.cc:359] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.124785 10052 raft_consensus.cc:385] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.125097 10052 raft_consensus.cc:740] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.125638 10054 tablet_bootstrap.cc:492] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:04.126075 10054 ts_tablet_manager.cc:1403] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.078s user 0.021s sys 0.000s
I20260430 07:58:04.128947 10054 raft_consensus.cc:359] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.125883 10052 consensus_queue.cc:260] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.129247 9760 catalog_manager.cc:5671] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:04.129746 10054 raft_consensus.cc:385] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.135752 10054 raft_consensus.cc:740] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.137166 10054 consensus_queue.cc:260] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.138850 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:04.139894 10054 ts_tablet_manager.cc:1434] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260430 07:58:04.140087 10011 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.141266 10054 tablet_bootstrap.cc:492] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:04.141911 10058 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.142544 10058 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.131588 10052 ts_tablet_manager.cc:1434] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.038s user 0.008s sys 0.007s
I20260430 07:58:04.132027 9863 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.146592 10052 tablet_bootstrap.cc:492] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:04.154278 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:04.154752 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:04.155162 9937 raft_consensus.cc:2393] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 1: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:04.182756 9863 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:04.188787 9904 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:04.185043 10058 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.190217 10053 tablet_bootstrap.cc:492] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:04.188915 10011 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:04.191255 10053 ts_tablet_manager.cc:1403] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.145s user 0.042s sys 0.021s
I20260430 07:58:04.190430 10054 tablet_bootstrap.cc:654] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.191699 9828 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:04.195374 10053 raft_consensus.cc:359] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.196086 10053 raft_consensus.cc:385] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.196436 10053 raft_consensus.cc:740] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.197057 10053 consensus_queue.cc:260] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.190267 10087 raft_consensus.cc:2804] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:04.198506 10087 raft_consensus.cc:697] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:04.195068 10052 tablet_bootstrap.cc:654] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.199689 10053 ts_tablet_manager.cc:1434] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260430 07:58:04.199385 10087 consensus_queue.cc:237] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.200598 10053 tablet_bootstrap.cc:492] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:04.201854 9759 catalog_manager.cc:5671] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:04.205097 10058 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.208922 10053 tablet_bootstrap.cc:654] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.227533 10058 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:04.228379 10054 tablet_bootstrap.cc:492] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:04.229090 10053 tablet_bootstrap.cc:492] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:04.229250 10052 tablet_bootstrap.cc:492] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:04.228854 10054 ts_tablet_manager.cc:1403] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.088s user 0.015s sys 0.009s
I20260430 07:58:04.229559 10053 ts_tablet_manager.cc:1403] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.000s
I20260430 07:58:04.229743 10052 ts_tablet_manager.cc:1403] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.083s user 0.020s sys 0.036s
I20260430 07:58:04.232262 10054 raft_consensus.cc:359] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.232537 10052 raft_consensus.cc:359] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.233029 10054 raft_consensus.cc:385] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.232806 10053 raft_consensus.cc:359] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.233250 10052 raft_consensus.cc:385] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.233371 10054 raft_consensus.cc:740] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.233565 10053 raft_consensus.cc:385] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.233784 10052 raft_consensus.cc:740] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.234079 10053 raft_consensus.cc:740] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.233366 9759 catalog_manager.cc:5671] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:04.234445 10054 consensus_queue.cc:260] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.236220 10052 consensus_queue.cc:260] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.236256 10053 consensus_queue.cc:260] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.238953 10053 ts_tablet_manager.cc:1434] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260430 07:58:04.239223 10052 ts_tablet_manager.cc:1434] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.009s user 0.003s sys 0.005s
I20260430 07:58:04.240239 10052 tablet_bootstrap.cc:492] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:04.242043 10053 tablet_bootstrap.cc:492] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:04.245976 10054 ts_tablet_manager.cc:1434] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.016s user 0.007s sys 0.001s
I20260430 07:58:04.246948 10054 tablet_bootstrap.cc:492] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:04.249795 10061 consensus_queue.cc:1048] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:04.253712 10061 consensus_queue.cc:1048] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:04.259193 10052 tablet_bootstrap.cc:654] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.263518 10053 tablet_bootstrap.cc:654] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.266362 10087 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:04.275125 10054 tablet_bootstrap.cc:654] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.282558 10061 consensus_queue.cc:1048] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:04.294179 10052 tablet_bootstrap.cc:492] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:04.294901 10052 ts_tablet_manager.cc:1403] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.055s user 0.015s sys 0.006s
I20260430 07:58:04.299475 10052 raft_consensus.cc:359] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.300235 10052 raft_consensus.cc:385] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.300547 10052 raft_consensus.cc:740] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.302779 10052 consensus_queue.cc:260] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.311866 10052 ts_tablet_manager.cc:1434] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.016s user 0.010s sys 0.000s
I20260430 07:58:04.326112 10122 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:04.327095 10080 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.327860 10053 tablet_bootstrap.cc:492] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:04.327675 10080 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.328311 10053 ts_tablet_manager.cc:1403] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.087s user 0.016s sys 0.011s
I20260430 07:58:04.328662 10052 tablet_bootstrap.cc:492] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:04.331451 10080 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:04.334654 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:04.336920 9863 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 1: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:04.331552 10053 raft_consensus.cc:359] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.338488 10053 raft_consensus.cc:385] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.338932 10053 raft_consensus.cc:740] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.341095 10052 tablet_bootstrap.cc:654] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.340123 10053 consensus_queue.cc:260] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.341117 10082 consensus_queue.cc:1048] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:04.345255 10053 ts_tablet_manager.cc:1434] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.017s user 0.006s sys 0.005s
I20260430 07:58:04.346371 10053 tablet_bootstrap.cc:492] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:04.348162 10054 tablet_bootstrap.cc:492] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:04.348673 10054 ts_tablet_manager.cc:1403] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.102s user 0.013s sys 0.024s
I20260430 07:58:04.352594 10054 raft_consensus.cc:359] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.353236 10053 tablet_bootstrap.cc:654] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.353392 10054 raft_consensus.cc:385] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.353705 10054 raft_consensus.cc:740] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.354712 10054 consensus_queue.cc:260] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.359719 10054 ts_tablet_manager.cc:1434] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260430 07:58:04.360992 10053 tablet_bootstrap.cc:492] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:04.362026 10053 ts_tablet_manager.cc:1403] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260430 07:58:04.361037 10054 tablet_bootstrap.cc:492] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:04.362603 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:04.364645 9935 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 1: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:04.367210 10052 tablet_bootstrap.cc:492] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:04.367898 10052 ts_tablet_manager.cc:1403] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.039s user 0.020s sys 0.006s
I20260430 07:58:04.388511 10052 raft_consensus.cc:359] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.389580 10052 raft_consensus.cc:385] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.390004 10052 raft_consensus.cc:740] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.390045 9977 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:04.391170 10082 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.392243 10053 raft_consensus.cc:359] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.392952 10053 raft_consensus.cc:385] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.393235 10053 raft_consensus.cc:740] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.403823 10054 tablet_bootstrap.cc:654] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.403482 10053 consensus_queue.cc:260] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.410557 10052 consensus_queue.cc:260] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.415838 10053 ts_tablet_manager.cc:1434] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20260430 07:58:04.420410 10052 ts_tablet_manager.cc:1434] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.052s user 0.008s sys 0.002s
I20260430 07:58:04.421904 10052 tablet_bootstrap.cc:492] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:04.423558 10087 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.424917 10087 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.428731 10122 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.429750 10122 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.433326 10082 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:04.434198 10087 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.434577 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:04.435480 9862 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:04.435911 10053 tablet_bootstrap.cc:492] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:04.437606 10122 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.440899 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:04.442606 10011 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:04.444932 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:04.445181 9903 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:04.445587 10012 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:04.446141 10122 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:04.446785 9904 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:04.447536 10122 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:04.447911 10122 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.448172 10087 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:04.448572 10087 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:04.448678 10053 tablet_bootstrap.cc:654] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.448985 10087 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.435504 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:04.450691 9863 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:04.441996 10052 tablet_bootstrap.cc:654] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.446416 10054 tablet_bootstrap.cc:492] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:04.460225 10054 ts_tablet_manager.cc:1403] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.099s user 0.027s sys 0.014s
I20260430 07:58:04.461690 10122 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.465204 10054 raft_consensus.cc:359] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.465993 10054 raft_consensus.cc:385] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.466224 10053 tablet_bootstrap.cc:492] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:04.466315 10054 raft_consensus.cc:740] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.465742 10087 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.466802 10053 ts_tablet_manager.cc:1403] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.007s
I20260430 07:58:04.467101 10054 consensus_queue.cc:260] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.467767 10058 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.468204 10058 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.470048 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:04.470463 10058 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.470675 9863 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.470853 10054 ts_tablet_manager.cc:1434] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20260430 07:58:04.471403 10052 tablet_bootstrap.cc:492] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:04.471832 10054 tablet_bootstrap.cc:492] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:04.472644 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:04.472728 10053 raft_consensus.cc:359] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.473333 10012 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:04.473526 10053 raft_consensus.cc:385] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.473839 10053 raft_consensus.cc:740] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.474500 10053 consensus_queue.cc:260] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.476583 10053 ts_tablet_manager.cc:1434] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260430 07:58:04.471854 10052 ts_tablet_manager.cc:1403] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.011s
I20260430 07:58:04.479275 10054 tablet_bootstrap.cc:654] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.479503 9828 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:04.480566 10052 raft_consensus.cc:359] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.481240 10052 raft_consensus.cc:385] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.481559 9863 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:04.481663 10058 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:04.482551 10058 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:04.482975 9904 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:04.487571 10058 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.488348 10101 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:04.488843 10101 raft_consensus.cc:697] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:04.489734 10101 consensus_queue.cc:237] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.483862 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:04.492071 9937 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:04.494825 10122 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.485208 10053 tablet_bootstrap.cc:492] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:04.481854 10052 raft_consensus.cc:740] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.496783 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:04.498590 10012 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.498704 10052 consensus_queue.cc:260] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.505906 10053 tablet_bootstrap.cc:654] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.508227 10113 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:04.509402 10122 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:04.511336 10012 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:04.513236 10052 ts_tablet_manager.cc:1434] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.036s user 0.002s sys 0.005s
I20260430 07:58:04.514328 10052 tablet_bootstrap.cc:492] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:04.504940 10058 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.526780 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:04.527489 9935 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.529565 10052 tablet_bootstrap.cc:654] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.530401 10053 tablet_bootstrap.cc:492] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:04.530869 10053 ts_tablet_manager.cc:1403] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.003s
I20260430 07:58:04.533553 10058 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.534125 10053 raft_consensus.cc:359] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.535262 10053 raft_consensus.cc:385] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.535562 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:04.536942 10011 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.536369 10054 tablet_bootstrap.cc:492] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:04.538458 10054 ts_tablet_manager.cc:1403] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.067s user 0.018s sys 0.015s
I20260430 07:58:04.542100 10054 raft_consensus.cc:359] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.542873 10054 raft_consensus.cc:385] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.543149 10054 raft_consensus.cc:740] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.544179 10054 consensus_queue.cc:260] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.535591 10053 raft_consensus.cc:740] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.550463 10061 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.550989 10061 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.551316 9935 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:04.554296 10087 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.554020 10080 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:04.554981 10087 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:04.555929 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:04.556660 9862 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.546375 10053 consensus_queue.cc:260] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.561841 10061 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:04.563409 9828 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:04.565241 10054 ts_tablet_manager.cc:1434] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20260430 07:58:04.559652 10053 ts_tablet_manager.cc:1434] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20260430 07:58:04.567167 10053 tablet_bootstrap.cc:492] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:04.567196 10054 tablet_bootstrap.cc:492] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:04.579540 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:04.567871 10058 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:04.589099 10058 raft_consensus.cc:697] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:04.590072 10058 consensus_queue.cc:237] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.593609 9862 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:04.593353 10011 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:04.620296 10087 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.609068 10060 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260430 07:58:04.626047 10087 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.631922 10087 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.626819 9904 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:04.602450 9978 leader_election.cc:343] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.635201 9761 catalog_manager.cc:5671] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:04.643976 10149 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:04.644655 10149 raft_consensus.cc:697] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:04.645040 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:04.645915 9860 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:04.647588 10052 tablet_bootstrap.cc:492] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:04.647459 9904 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:04.648031 10052 ts_tablet_manager.cc:1403] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.134s user 0.027s sys 0.026s
I20260430 07:58:04.648295 10060 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:04.648787 10060 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:04.649103 10060 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.590370 10113 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.035s
I20260430 07:58:04.656661 10052 raft_consensus.cc:359] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.657707 10052 raft_consensus.cc:385] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.658031 10052 raft_consensus.cc:740] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.659080 10052 consensus_queue.cc:260] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.661552 10052 ts_tablet_manager.cc:1434] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260430 07:58:04.663036 10052 tablet_bootstrap.cc:492] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:04.665066 10153 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.665616 10153 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:04.666811 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 135849us
I20260430 07:58:04.669713 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:04.668619 10149 consensus_queue.cc:237] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.672812 10122 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.673357 10122 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.672250 10153 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.675536 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:04.676409 10011 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.677752 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:04.679481 9828 leader_election.cc:343] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.675956 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:04.683455 9828 leader_election.cc:343] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.684092 9828 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:04.687729 9761 catalog_manager.cc:5671] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:04.694842 10053 tablet_bootstrap.cc:654] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.693333 10122 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.704447 10054 tablet_bootstrap.cc:654] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.705746 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:04.700429 10153 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:04.717108 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:04.720379 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:04.723609 9977 leader_election.cc:343] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
W20260430 07:58:04.724108 9904 leader_election.cc:343] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
W20260430 07:58:04.725662 9903 leader_election.cc:343] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.724215 9977 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:04.727660 10082 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:04.727686 10061 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:04.730553 10148 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260430 07:58:04.732887 10087 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.733548 10087 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.735302 10087 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.688280 10060 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:04.731473 9903 leader_election.cc:343] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.740103 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:04.745221 9903 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:04.746546 10087 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260430 07:58:04.746759 9904 leader_election.cc:343] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.752918 10060 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.753947 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:04.756223 9863 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.778033 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:04.781971 10052 tablet_bootstrap.cc:654] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.793570 10111 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.798372 10054 tablet_bootstrap.cc:492] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:04.800284 10054 ts_tablet_manager.cc:1403] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.233s user 0.018s sys 0.014s
W20260430 07:58:04.798751 9903 leader_election.cc:343] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.797813 10111 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.801620 9903 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:04.803267 10111 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:04.803983 10060 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:04.804986 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:04.806645 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:04.808571 9978 leader_election.cc:343] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.813117 9863 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:04.815461 9904 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:04.817058 10149 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:04.817896 10149 raft_consensus.cc:697] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:04.819408 10149 consensus_queue.cc:237] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.828493 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:04.833415 9977 leader_election.cc:343] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.834108 9977 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:04.836048 10111 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260430 07:58:04.842046 9903 leader_election.cc:343] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.851511 10011 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:04.855213 10052 tablet_bootstrap.cc:492] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:04.856021 10052 ts_tablet_manager.cc:1403] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.193s user 0.009s sys 0.007s
I20260430 07:58:04.848316 10054 raft_consensus.cc:359] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.859881 10054 raft_consensus.cc:385] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.860494 10054 raft_consensus.cc:740] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.861572 10054 consensus_queue.cc:260] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.864307 10053 tablet_bootstrap.cc:492] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:04.864459 10054 ts_tablet_manager.cc:1434] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.064s user 0.012s sys 0.012s
I20260430 07:58:04.864878 10053 ts_tablet_manager.cc:1403] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.298s user 0.021s sys 0.034s
I20260430 07:58:04.865741 10054 tablet_bootstrap.cc:492] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:04.868139 10053 raft_consensus.cc:359] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.868561 10052 raft_consensus.cc:359] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.870298 10052 raft_consensus.cc:385] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.870639 10052 raft_consensus.cc:740] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.871578 10052 consensus_queue.cc:260] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.872900 10054 tablet_bootstrap.cc:654] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.875142 10053 raft_consensus.cc:385] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.875767 10053 raft_consensus.cc:740] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.876590 10053 consensus_queue.cc:260] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.880540 10053 ts_tablet_manager.cc:1434] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.015s user 0.010s sys 0.001s
I20260430 07:58:04.881515 10053 tablet_bootstrap.cc:492] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:04.882202 10052 ts_tablet_manager.cc:1434] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.026s user 0.006s sys 0.005s
I20260430 07:58:04.881731 9759 catalog_manager.cc:5671] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:04.885846 10054 tablet_bootstrap.cc:492] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:04.886360 10054 ts_tablet_manager.cc:1403] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.012s
I20260430 07:58:04.887089 10052 tablet_bootstrap.cc:492] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:04.892091 10053 tablet_bootstrap.cc:654] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.894709 10153 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.895273 10153 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.896337 10054 raft_consensus.cc:359] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.897714 10054 raft_consensus.cc:385] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.898005 10054 raft_consensus.cc:740] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.899302 10054 consensus_queue.cc:260] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.899824 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:04.901945 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:04.903893 9828 leader_election.cc:343] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.904232 10054 ts_tablet_manager.cc:1434] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.018s user 0.008s sys 0.002s
I20260430 07:58:04.905766 10054 tablet_bootstrap.cc:492] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:04.905910 10061 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.906598 10061 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:04.909726 9828 leader_election.cc:343] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.910229 9828 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:04.911623 10113 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:04.911455 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:04.912397 10153 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:04.913048 10054 tablet_bootstrap.cc:654] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
W20260430 07:58:04.913791 9978 leader_election.cc:343] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:04.920049 10061 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:04.920857 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:04.921631 9937 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:04.923115 9977 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:04.924360 10061 raft_consensus.cc:2804] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:04.925073 10061 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:04.926422 10061 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:04.928579 10052 tablet_bootstrap.cc:654] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.932178 10149 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:04.950628 9760 catalog_manager.cc:5671] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:04.960294 10053 tablet_bootstrap.cc:492] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:04.960973 10053 ts_tablet_manager.cc:1403] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.080s user 0.030s sys 0.010s
I20260430 07:58:04.961035 10052 tablet_bootstrap.cc:492] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:04.961922 10052 ts_tablet_manager.cc:1403] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.075s user 0.016s sys 0.006s
I20260430 07:58:04.967212 10148 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:04.971333 10052 raft_consensus.cc:359] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.972740 10054 tablet_bootstrap.cc:492] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:04.973237 10054 ts_tablet_manager.cc:1403] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.068s user 0.017s sys 0.018s
I20260430 07:58:04.973587 10053 raft_consensus.cc:359] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.974913 10053 raft_consensus.cc:385] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.975749 10053 raft_consensus.cc:740] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.972146 10052 raft_consensus.cc:385] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.976665 10052 raft_consensus.cc:740] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.977451 10052 consensus_queue.cc:260] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.981853 10053 consensus_queue.cc:260] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.982959 10054 raft_consensus.cc:359] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.983968 10054 raft_consensus.cc:385] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:04.984330 10054 raft_consensus.cc:740] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:04.984844 10053 ts_tablet_manager.cc:1434] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.023s user 0.006s sys 0.003s
I20260430 07:58:04.985257 10054 consensus_queue.cc:260] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.985879 10053 tablet_bootstrap.cc:492] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:04.987032 10052 ts_tablet_manager.cc:1434] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.025s user 0.005s sys 0.003s
I20260430 07:58:04.987411 10054 ts_tablet_manager.cc:1434] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260430 07:58:04.988096 10052 tablet_bootstrap.cc:492] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:04.988247 10054 tablet_bootstrap.cc:492] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:04.990702 10153 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:04.991359 10153 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:04.994854 10053 tablet_bootstrap.cc:654] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.995558 10054 tablet_bootstrap.cc:654] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:04.998906 10153 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.002336 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.005787 10010 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:05.006275 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:05.008100 9828 leader_election.cc:343] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.009811 10052 tablet_bootstrap.cc:654] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.010013 9828 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.011303 10153 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:05.011714 10153 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:05.012058 10153 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:05.017138 9809 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:58:05.021255 10149 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.021857 10149 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.023355 10052 tablet_bootstrap.cc:492] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:05.024575 10052 ts_tablet_manager.cc:1403] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.010s
I20260430 07:58:05.025811 10149 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.027732 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.028626 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:05.029089 9904 leader_election.cc:343] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.028846 10052 raft_consensus.cc:359] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.029645 10052 raft_consensus.cc:385] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.029969 10052 raft_consensus.cc:740] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.025863 10061 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:05.030231 9903 leader_election.cc:343] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.030846 9903 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:05.030802 10052 consensus_queue.cc:260] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.031754 10149 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:05.034157 10052 ts_tablet_manager.cc:1434] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260430 07:58:05.034984 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:05.035760 9861 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.036177 10052 tablet_bootstrap.cc:492] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:05.043862 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:05.044989 9937 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.053323 9861 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:05.058108 9937 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:05.059019 10061 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:05.062323 9978 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:05.064759 10052 tablet_bootstrap.cc:654] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.069099 10061 raft_consensus.cc:2804] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:05.070364 10061 raft_consensus.cc:697] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:05.071197 10061 consensus_queue.cc:237] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.072652 10054 tablet_bootstrap.cc:492] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:05.065994 10153 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.066220 10082 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.068248 10053 tablet_bootstrap.cc:492] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:05.074018 10082 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.074344 10053 ts_tablet_manager.cc:1403] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.089s user 0.013s sys 0.001s
I20260430 07:58:05.076601 10153 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.076538 10082 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:05.080188 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.082790 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.077414 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:05.083755 9977 leader_election.cc:343] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.084022 10011 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:05.084980 9978 leader_election.cc:343] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.085414 9978 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.086392 10052 tablet_bootstrap.cc:492] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:05.086862 10052 ts_tablet_manager.cc:1403] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.051s user 0.018s sys 0.000s
I20260430 07:58:05.086948 10082 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:05.089780 10052 raft_consensus.cc:359] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.090619 10052 raft_consensus.cc:385] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.090896 10052 raft_consensus.cc:740] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.078619 10053 raft_consensus.cc:359] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.091709 10053 raft_consensus.cc:385] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.092084 10053 raft_consensus.cc:740] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.091528 10052 consensus_queue.cc:260] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.076552 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:05.092995 10053 consensus_queue.cc:260] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.093655 10011 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
W20260430 07:58:05.094159 9828 leader_election.cc:343] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.091167 9760 catalog_manager.cc:5671] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:05.094938 9828 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.095996 10153 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:05.093547 10152 consensus_queue.cc:1048] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:05.097121 10153 raft_consensus.cc:697] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:05.098453 10153 consensus_queue.cc:237] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.099663 10053 ts_tablet_manager.cc:1434] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20260430 07:58:05.073223 10054 ts_tablet_manager.cc:1403] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.085s user 0.015s sys 0.007s
I20260430 07:58:05.112411 10152 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.113157 10152 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.115777 10052 ts_tablet_manager.cc:1434] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260430 07:58:05.136404 10052 tablet_bootstrap.cc:492] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:05.136858 10054 raft_consensus.cc:359] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.137686 10054 raft_consensus.cc:385] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.137603 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.138737 10054 raft_consensus.cc:740] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
W20260430 07:58:05.140391 9828 leader_election.cc:343] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.141490 10053 tablet_bootstrap.cc:492] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:05.141331 10054 consensus_queue.cc:260] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.146759 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.147431 10011 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:05.148361 10054 ts_tablet_manager.cc:1434] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.041s user 0.009s sys 0.006s
I20260430 07:58:05.149516 10054 tablet_bootstrap.cc:492] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:05.149729 10058 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.150309 10058 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.150513 9828 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.152015 10113 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:05.152400 10113 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:05.152781 10113 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.154302 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.155375 9935 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:05.152407 10058 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.155498 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.165432 10113 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.168401 10053 tablet_bootstrap.cc:654] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.168540 10113 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:05.170217 9828 leader_election.cc:343] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.171006 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:05.177381 10061 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.152015 10152 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.176064 10153 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:05.180171 10153 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:05.180608 10153 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.177934 10061 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.181803 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:05.182621 10061 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:05.183996 9828 leader_election.cc:343] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.184490 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.185088 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.185755 9935 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
W20260430 07:58:05.186056 9978 leader_election.cc:343] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.187352 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:05.188102 10011 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.173087 9758 catalog_manager.cc:5671] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:05.194442 9977 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:05.197858 10061 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:05.198438 10061 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:05.198853 10061 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.202013 10152 consensus_queue.cc:1048] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260430 07:58:05.210175 10153 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.211758 10052 tablet_bootstrap.cc:654] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.192579 10179 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.213860 10179 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.217159 10153 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.217332 10149 consensus_queue.cc:1048] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:05.225773 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:05.226054 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:05.235980 10053 tablet_bootstrap.cc:492] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:05.236603 10053 ts_tablet_manager.cc:1403] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.095s user 0.014s sys 0.026s
I20260430 07:58:05.224725 10011 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:05.236136 9937 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.242844 10179 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.243454 10053 raft_consensus.cc:359] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.244692 10053 raft_consensus.cc:385] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.245003 10053 raft_consensus.cc:740] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.245761 10053 consensus_queue.cc:260] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.248229 10061 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:05.247232 9828 leader_election.cc:343] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.251029 10061 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:05.252210 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:05.252425 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.252682 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.253086 9935 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:05.253317 10011 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
W20260430 07:58:05.254014 9978 leader_election.cc:343] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.256248 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:05.258793 9935 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.259680 9937 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:05.252269 9828 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.266688 10179 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:05.267367 10179 raft_consensus.cc:697] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:05.268487 10179 consensus_queue.cc:237] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.280154 10053 ts_tablet_manager.cc:1434] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.043s user 0.007s sys 0.006s
I20260430 07:58:05.283995 10053 tablet_bootstrap.cc:492] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
W20260430 07:58:05.294874 9903 consensus_peers.cc:597] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 -> Peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Couldn't send request to peer 48e81934d52a4e93be5171ceea193032. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:05.307196 10053 tablet_bootstrap.cc:654] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.318137 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:05.318986 10054 tablet_bootstrap.cc:654] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.319742 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:05.324000 10153 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:05.324651 10179 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:05.324704 10153 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:05.325412 10153 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:05.326095 10053 tablet_bootstrap.cc:492] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:05.327008 10053 ts_tablet_manager.cc:1403] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.002s
I20260430 07:58:05.327903 10179 raft_consensus.cc:697] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:05.329209 10061 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.329849 10061 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.331861 10061 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:05.336606 10052 tablet_bootstrap.cc:492] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:05.336179 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.328824 10179 consensus_queue.cc:237] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.339596 9935 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:05.337129 10052 ts_tablet_manager.cc:1403] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.201s user 0.036s sys 0.034s
I20260430 07:58:05.336436 10053 raft_consensus.cc:359] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.341256 10053 raft_consensus.cc:385] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.341657 10053 raft_consensus.cc:740] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.342955 10053 consensus_queue.cc:260] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.343461 10052 raft_consensus.cc:359] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.344861 10053 ts_tablet_manager.cc:1434] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260430 07:58:05.344513 10052 raft_consensus.cc:385] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260430 07:58:05.345695 9977 leader_election.cc:343] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.347124 10087 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.346951 9977 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:05.347642 10087 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.348704 10082 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:05.349332 10082 raft_consensus.cc:697] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:05.349454 10122 consensus_queue.cc:1048] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:05.350566 10082 consensus_queue.cc:237] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.351089 10052 raft_consensus.cc:740] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.351186 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.351967 9861 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:05.353842 10053 tablet_bootstrap.cc:492] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:05.353613 9978 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.353958 10052 consensus_queue.cc:260] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.355762 10111 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:05.356153 10111 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:05.356976 10111 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.357692 10052 ts_tablet_manager.cc:1434] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260430 07:58:05.361490 10052 tablet_bootstrap.cc:492] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:05.361963 10087 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.363106 10053 tablet_bootstrap.cc:654] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.362918 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.350534 10153 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.367153 10153 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.367825 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.363967 9861 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:05.370605 10113 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.371254 10113 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.373236 10113 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.375842 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:05.380340 9828 leader_election.cc:343] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.377530 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:05.381481 10011 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:05.373744 9758 catalog_manager.cc:5671] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
W20260430 07:58:05.375344 9903 leader_election.cc:343] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.380954 10111 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.391103 10052 tablet_bootstrap.cc:654] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.390473 9904 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:05.392730 10087 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:05.393335 10087 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:05.397236 10087 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.400173 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:05.400790 9861 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.405176 10111 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:05.396569 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:05.409323 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.410691 10053 tablet_bootstrap.cc:492] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:05.414002 10053 ts_tablet_manager.cc:1403] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.060s user 0.016s sys 0.008s
I20260430 07:58:05.414279 9937 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:05.419816 9828 leader_election.cc:343] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.419970 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:05.420425 9828 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.422003 10011 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:05.421790 10087 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.422894 10153 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:05.425138 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:05.425606 10087 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.425791 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:05.426188 10153 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:05.426352 9863 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.426725 10153 raft_consensus.cc:697] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:05.427346 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:05.427610 10153 consensus_queue.cc:237] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:05.428635 9903 leader_election.cc:343] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
W20260430 07:58:05.431478 9977 leader_election.cc:343] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.431582 10053 raft_consensus.cc:359] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.433146 10053 raft_consensus.cc:385] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.433527 10053 raft_consensus.cc:740] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.438827 10053 consensus_queue.cc:260] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.436379 9937 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:05.441452 10053 ts_tablet_manager.cc:1434] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260430 07:58:05.442395 10053 tablet_bootstrap.cc:492] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:05.449935 9861 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:05.453109 10149 consensus_queue.cc:1048] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:05.453251 9978 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.456468 9863 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:05.461879 10111 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:05.550314 10052 tablet_bootstrap.cc:492] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:05.551039 10052 ts_tablet_manager.cc:1403] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.191s user 0.025s sys 0.011s
I20260430 07:58:05.551155 10054 tablet_bootstrap.cc:492] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:05.551643 10054 ts_tablet_manager.cc:1403] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.402s user 0.024s sys 0.004s
I20260430 07:58:05.558725 10052 raft_consensus.cc:359] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.558594 10054 raft_consensus.cc:359] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.559412 10054 raft_consensus.cc:385] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.559696 10054 raft_consensus.cc:740] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.560284 10052 raft_consensus.cc:385] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.560598 10052 raft_consensus.cc:740] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.560822 10054 consensus_queue.cc:260] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.561448 10052 consensus_queue.cc:260] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.563053 10054 ts_tablet_manager.cc:1434] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20260430 07:58:05.555943 10111 raft_consensus.cc:697] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:05.563637 10052 ts_tablet_manager.cc:1434] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260430 07:58:05.563963 10054 tablet_bootstrap.cc:492] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:05.564536 10052 tablet_bootstrap.cc:492] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:05.564102 10111 consensus_queue.cc:237] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:05.518412 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 57562.2us
I20260430 07:58:05.488626 9759 catalog_manager.cc:5671] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:05.570040 9904 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:05.573784 10122 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:05.573971 10148 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.578243 10122 raft_consensus.cc:697] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:05.578342 10148 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.579177 10122 consensus_queue.cc:237] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.586714 9759 catalog_manager.cc:5671] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 2, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 2 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:05.603325 10148 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.607146 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.608981 10010 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:05.611047 9903 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:05.603149 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.614217 10060 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:05.615343 10060 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:05.616240 10060 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:05.616390 9904 leader_election.cc:343] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.617450 9937 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.616459 10061 consensus_queue.cc:1048] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260430 07:58:05.630465 10053 tablet_bootstrap.cc:654] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.630513 9760 catalog_manager.cc:5671] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:05.641458 9758 catalog_manager.cc:5671] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:05.645398 10060 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.651404 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:05.653707 10054 tablet_bootstrap.cc:654] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
W20260430 07:58:05.653553 9904 leader_election.cc:343] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.656342 10082 consensus_queue.cc:1048] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:05.660425 10060 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.663555 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:05.664702 10010 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.669142 10179 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:05.671452 10054 tablet_bootstrap.cc:492] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:05.671918 10054 ts_tablet_manager.cc:1403] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.108s user 0.010s sys 0.003s
I20260430 07:58:05.674505 10054 raft_consensus.cc:359] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.675040 10054 raft_consensus.cc:385] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.675280 10054 raft_consensus.cc:740] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.675863 10054 consensus_queue.cc:260] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.677608 10054 ts_tablet_manager.cc:1434] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260430 07:58:05.678515 10054 tablet_bootstrap.cc:492] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:05.679728 10052 tablet_bootstrap.cc:654] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.695889 10010 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:05.707967 10053 tablet_bootstrap.cc:492] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:05.708655 10053 ts_tablet_manager.cc:1403] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.266s user 0.008s sys 0.014s
I20260430 07:58:05.708614 9903 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:05.709488 10149 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:05.711699 10060 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:05.711932 10149 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.712405 10060 raft_consensus.cc:697] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:05.716143 10060 consensus_queue.cc:237] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.718711 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.718847 10149 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.722057 9760 catalog_manager.cc:5671] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:05.721817 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.729658 10010 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:05.730702 9903 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:05.723129 10054 tablet_bootstrap.cc:654] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.732813 10149 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:05.733734 10149 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:05.735229 10149 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:05.741472 10052 tablet_bootstrap.cc:492] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:05.742300 10052 ts_tablet_manager.cc:1403] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.178s user 0.010s sys 0.009s
I20260430 07:58:05.743542 10113 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:05.753275 10052 raft_consensus.cc:359] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.753405 10053 raft_consensus.cc:359] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.754076 10052 raft_consensus.cc:385] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.755358 10052 raft_consensus.cc:740] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.754245 10053 raft_consensus.cc:385] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.756477 10052 consensus_queue.cc:260] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.759048 10052 ts_tablet_manager.cc:1434] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260430 07:58:05.761497 10052 tablet_bootstrap.cc:492] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:05.761236 10149 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.756829 10053 raft_consensus.cc:740] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.765143 10053 consensus_queue.cc:260] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.767253 10054 tablet_bootstrap.cc:492] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:05.767746 10054 ts_tablet_manager.cc:1403] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.006s
I20260430 07:58:05.767798 10053 ts_tablet_manager.cc:1434] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.058s user 0.009s sys 0.000s
I20260430 07:58:05.769485 10149 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.770645 10053 tablet_bootstrap.cc:492] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:05.769814 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:05.772408 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:05.773581 10111 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:05.774040 9903 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:05.776866 9760 catalog_manager.cc:5671] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:05.777019 10060 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260430 07:58:05.788434 9978 consensus_peers.cc:597] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 -> Peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Couldn't send request to peer 27f97a330f374849be58a168318c1ec6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:05.792701 10053 tablet_bootstrap.cc:654] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.795562 10149 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.797617 10054 raft_consensus.cc:359] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.798755 10052 tablet_bootstrap.cc:654] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.798800 10054 raft_consensus.cc:385] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.796943 10149 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.803495 10149 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.801234 10054 raft_consensus.cc:740] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.804404 10054 consensus_queue.cc:260] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.806880 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.807669 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.808444 10010 raft_consensus.cc:2393] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 1: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:05.809079 10149 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.809608 10149 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.811646 10149 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.812672 10053 tablet_bootstrap.cc:492] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:05.813885 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.814908 9860 raft_consensus.cc:2393] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 1: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:05.815426 10053 ts_tablet_manager.cc:1403] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.007s
I20260430 07:58:05.807090 10054 ts_tablet_manager.cc:1434] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.039s user 0.006s sys 0.004s
I20260430 07:58:05.817032 9903 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:05.820398 10060 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.821033 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.823906 10054 tablet_bootstrap.cc:492] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:05.824927 10052 tablet_bootstrap.cc:492] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:05.825448 10052 ts_tablet_manager.cc:1403] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.064s user 0.013s sys 0.007s
I20260430 07:58:05.826871 9903 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:05.828390 10053 raft_consensus.cc:359] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.829751 10053 raft_consensus.cc:385] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.830000 10053 raft_consensus.cc:740] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.830837 10053 consensus_queue.cc:260] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.833451 10053 ts_tablet_manager.cc:1434] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.018s user 0.007s sys 0.001s
I20260430 07:58:05.834403 10053 tablet_bootstrap.cc:492] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:05.836139 10149 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.837244 10060 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:05.839566 10052 raft_consensus.cc:359] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.840337 10052 raft_consensus.cc:385] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.840425 10054 tablet_bootstrap.cc:654] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.840648 10052 raft_consensus.cc:740] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.841383 10052 consensus_queue.cc:260] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.844798 10149 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:05.847299 10113 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.847832 10113 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.853760 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.857724 10011 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260430 07:58:05.857622 10052 ts_tablet_manager.cc:1434] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.032s user 0.007s sys 0.001s
I20260430 07:58:05.856091 10054 tablet_bootstrap.cc:492] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:05.857206 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.850059 10113 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.847945 10053 tablet_bootstrap.cc:654] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
W20260430 07:58:05.860924 9828 leader_election.cc:343] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.863037 10054 ts_tablet_manager.cc:1403] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.004s
I20260430 07:58:05.864058 10052 tablet_bootstrap.cc:492] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
W20260430 07:58:05.864904 9828 leader_election.cc:343] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.865646 9828 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.867949 10113 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:05.868543 10054 raft_consensus.cc:359] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.869599 10054 raft_consensus.cc:385] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.869408 10113 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:05.869972 10054 raft_consensus.cc:740] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.871202 10052 tablet_bootstrap.cc:654] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.870916 10054 consensus_queue.cc:260] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.878566 10113 consensus_queue.cc:1048] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:05.888073 10054 ts_tablet_manager.cc:1434] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260430 07:58:05.889178 10054 tablet_bootstrap.cc:492] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:05.890483 10082 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.891000 10082 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.893505 10082 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:05.894305 9828 consensus_peers.cc:597] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 -> Peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Couldn't send request to peer 48e81934d52a4e93be5171ceea193032. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:05.894476 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.901813 10082 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.902878 10082 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.903712 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.905514 10179 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.905682 9977 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.906067 10179 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.907003 10061 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:05.908075 10179 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.909080 10082 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:05.908934 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:05.911589 9860 raft_consensus.cc:2393] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 1: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:05.913009 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.916930 9977 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.910514 10179 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.918327 10082 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.919054 10179 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.921247 10179 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:05.924434 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.925105 10009 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:05.927585 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.926564 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:05.931958 10010 raft_consensus.cc:2393] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 1: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:05.939546 10153 consensus_queue.cc:1048] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:05.942153 10054 tablet_bootstrap.cc:654] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:05.945655 10082 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:05.945834 9828 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:05.945705 10153 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:05.945469 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.949047 10113 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:05.949616 10113 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:05.949363 9938 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:05.950035 10113 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.952690 9828 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:05.953536 10153 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:05.978310 10082 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:05.979213 10052 tablet_bootstrap.cc:492] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:05.980659 10053 tablet_bootstrap.cc:492] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:05.982642 10053 ts_tablet_manager.cc:1403] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.148s user 0.018s sys 0.010s
I20260430 07:58:05.979499 10082 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.987470 10082 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:05.987514 10052 ts_tablet_manager.cc:1403] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.124s user 0.037s sys 0.021s
I20260430 07:58:05.990762 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:05.992709 9978 leader_election.cc:343] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:05.993624 10052 raft_consensus.cc:359] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.994254 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:05.994333 10053 raft_consensus.cc:359] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.995739 9937 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:05.995746 10053 raft_consensus.cc:385] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:05.997603 10053 raft_consensus.cc:740] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:05.997882 9977 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:05.999542 10082 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:05.999209 10053 consensus_queue.cc:260] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:05.999936 10082 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:06.001199 10082 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.001410 10052 raft_consensus.cc:385] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.001761 10052 raft_consensus.cc:740] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.002080 10053 ts_tablet_manager.cc:1434] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.019s user 0.009s sys 0.000s
I20260430 07:58:06.002414 10052 consensus_queue.cc:260] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.003103 10053 tablet_bootstrap.cc:492] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:06.003551 10113 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.005542 10113 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.006954 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:06.007620 10052 ts_tablet_manager.cc:1434] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260430 07:58:06.007579 9937 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.008757 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:06.009372 10010 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.013011 10052 tablet_bootstrap.cc:492] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:06.015399 9937 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:06.016853 9828 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:06.017571 10010 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:06.023180 10082 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.024041 10053 tablet_bootstrap.cc:654] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.025470 10113 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:06.026486 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:06.026646 10113 raft_consensus.cc:697] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:06.027585 10052 tablet_bootstrap.cc:654] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
W20260430 07:58:06.027812 9978 leader_election.cc:343] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.029968 10082 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:06.031570 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:06.035811 9937 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.037744 10052 tablet_bootstrap.cc:492] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:06.038245 10052 ts_tablet_manager.cc:1403] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20260430 07:58:06.040882 10153 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:06.046005 9937 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:06.028009 10113 consensus_queue.cc:237] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.047492 9977 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:06.048839 10082 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:06.049655 10082 raft_consensus.cc:697] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:06.050912 10054 tablet_bootstrap.cc:492] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:06.050454 10082 consensus_queue.cc:237] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.051393 10054 ts_tablet_manager.cc:1403] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.162s user 0.031s sys 0.039s
I20260430 07:58:06.041476 10052 raft_consensus.cc:359] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.053476 10052 raft_consensus.cc:385] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.053781 10052 raft_consensus.cc:740] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.054327 10054 raft_consensus.cc:359] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.054416 10052 consensus_queue.cc:260] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.055124 10054 raft_consensus.cc:385] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.055413 10054 raft_consensus.cc:740] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.056164 10054 consensus_queue.cc:260] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.056733 10052 ts_tablet_manager.cc:1434] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260430 07:58:06.059587 10060 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.060290 10060 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.066514 10052 tablet_bootstrap.cc:492] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:06.066738 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:06.068867 9903 leader_election.cc:343] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.069689 10060 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.070859 10054 ts_tablet_manager.cc:1434] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.019s user 0.007s sys 0.002s
I20260430 07:58:06.072402 10054 tablet_bootstrap.cc:492] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:06.075217 10053 tablet_bootstrap.cc:492] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:06.075707 10053 ts_tablet_manager.cc:1403] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.073s user 0.015s sys 0.000s
I20260430 07:58:06.078405 10053 raft_consensus.cc:359] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.079161 10053 raft_consensus.cc:385] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.079486 10053 raft_consensus.cc:740] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.080142 10053 consensus_queue.cc:260] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.083930 10053 ts_tablet_manager.cc:1434] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260430 07:58:06.084867 10053 tablet_bootstrap.cc:492] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:06.086575 10054 tablet_bootstrap.cc:654] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.087075 10052 tablet_bootstrap.cc:654] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.102514 10054 tablet_bootstrap.cc:492] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:06.103154 10054 ts_tablet_manager.cc:1403] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.012s
I20260430 07:58:06.107993 10053 tablet_bootstrap.cc:654] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.107898 10054 raft_consensus.cc:359] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.110813 10054 raft_consensus.cc:385] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.113333 10054 raft_consensus.cc:740] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
W20260430 07:58:06.113870 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 72745.2us
I20260430 07:58:06.117028 10060 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.119596 10054 consensus_queue.cc:260] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.119820 10060 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.121856 10054 ts_tablet_manager.cc:1434] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260430 07:58:06.122362 10060 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.122959 10054 tablet_bootstrap.cc:492] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:06.123936 10061 consensus_queue.cc:1048] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:06.126276 9937 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260430 07:58:06.128008 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:06.111994 9759 catalog_manager.cc:5671] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
W20260430 07:58:06.133086 9903 leader_election.cc:343] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.135154 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:06.135953 9861 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:06.136852 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:06.138289 10218 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.138926 10218 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.140909 10218 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.142793 10052 tablet_bootstrap.cc:492] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:06.143265 10052 ts_tablet_manager.cc:1403] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.077s user 0.017s sys 0.002s
I20260430 07:58:06.134435 10060 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260430 07:58:06.146258 10054 tablet_bootstrap.cc:654] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.148648 9761 catalog_manager.cc:5671] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:06.148993 10219 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260430 07:58:06.159394 9904 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:06.161123 10149 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:06.161558 10149 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:06.161917 10149 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.163220 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:06.164125 10052 raft_consensus.cc:359] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.164803 10052 raft_consensus.cc:385] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.165503 10052 raft_consensus.cc:740] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.166206 10052 consensus_queue.cc:260] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.167323 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:06.168135 9938 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:06.168908 10052 ts_tablet_manager.cc:1434] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20260430 07:58:06.171697 10052 tablet_bootstrap.cc:492] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:06.164140 10011 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:06.174880 10082 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:06.181838 9904 leader_election.cc:343] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.181936 9828 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:06.182364 9904 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:06.181869 10082 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.184146 10219 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:06.184701 10219 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:06.185055 10219 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.186493 10082 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:06.191583 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:06.186724 10060 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:06.219913 10149 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.227052 10218 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.236980 10053 tablet_bootstrap.cc:492] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:06.237854 10011 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.239439 10053 ts_tablet_manager.cc:1403] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.155s user 0.024s sys 0.010s
I20260430 07:58:06.229882 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:06.235246 9809 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:58:06.232265 10060 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260430 07:58:06.244663 10218 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:06.261817 9977 leader_election.cc:343] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.263214 10053 raft_consensus.cc:359] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.265101 10054 tablet_bootstrap.cc:492] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:06.266987 10053 raft_consensus.cc:385] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.267457 10053 raft_consensus.cc:740] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.267542 10054 ts_tablet_manager.cc:1403] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.145s user 0.037s sys 0.038s
I20260430 07:58:06.277532 10053 consensus_queue.cc:260] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.296934 10009 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:06.279249 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 94097.5us
I20260430 07:58:06.320721 10218 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.321727 10111 consensus_queue.cc:1048] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:06.325326 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:06.332028 10053 ts_tablet_manager.cc:1434] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.092s user 0.023s sys 0.034s
I20260430 07:58:06.333391 10053 tablet_bootstrap.cc:492] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:06.329548 10149 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.349083 9935 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:06.353055 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:06.339656 9903 leader_election.cc:343] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
W20260430 07:58:06.373696 9978 leader_election.cc:343] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.376919 9978 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:06.377521 10219 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.384850 10231 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:06.396624 10054 raft_consensus.cc:359] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.398761 10054 raft_consensus.cc:385] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.399168 10054 raft_consensus.cc:740] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.401834 10219 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.403424 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:06.403638 10054 consensus_queue.cc:260] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.407557 10007 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:06.404531 10233 log.cc:927] Time spent T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.065s user 0.000s sys 0.004s
I20260430 07:58:06.414309 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:06.431098 10054 ts_tablet_manager.cc:1434] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.161s user 0.006s sys 0.005s
I20260430 07:58:06.432439 10054 tablet_bootstrap.cc:492] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:06.436466 10052 tablet_bootstrap.cc:654] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.441790 9861 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.441608 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:06.446014 9863 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.454147 10219 consensus_queue.cc:1059] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:06.456048 10219 raft_consensus.cc:3055] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:06.456487 10219 raft_consensus.cc:740] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:06.460697 10053 tablet_bootstrap.cc:654] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.463682 10149 consensus_queue.cc:1048] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:06.457247 10219 consensus_queue.cc:260] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.466830 10219 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:06.467039 10149 consensus_queue.cc:1048] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:06.483037 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:06.487354 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:06.487764 9938 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.489027 10007 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:06.500679 10232 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.501521 10232 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.510825 10232 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.517043 9828 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:06.517987 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:06.519019 10007 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:06.525049 10053 tablet_bootstrap.cc:492] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:06.526350 10053 ts_tablet_manager.cc:1403] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.193s user 0.023s sys 0.006s
I20260430 07:58:06.529817 9938 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:06.534284 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:06.534826 9903 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:06.537186 10052 tablet_bootstrap.cc:492] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:06.537757 10052 ts_tablet_manager.cc:1403] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.366s user 0.019s sys 0.002s
I20260430 07:58:06.538998 10234 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:06.527534 10218 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:06.546247 10218 raft_consensus.cc:697] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:06.547070 10218 consensus_queue.cc:237] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.553647 10234 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:06.554165 10234 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.565644 10054 tablet_bootstrap.cc:654] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.528056 9863 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:06.576179 10052 raft_consensus.cc:359] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.577845 10052 raft_consensus.cc:385] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.578207 10052 raft_consensus.cc:740] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.579214 10052 consensus_queue.cc:260] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:06.574575 9904 leader_election.cc:343] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.591879 10234 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.594897 10052 ts_tablet_manager.cc:1434] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.057s user 0.006s sys 0.004s
I20260430 07:58:06.595963 10053 raft_consensus.cc:359] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.596791 10053 raft_consensus.cc:385] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.597182 10053 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.598054 10053 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.598660 10153 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:06.607734 10113 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:06.609489 10234 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.616803 9904 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:06.624779 10052 tablet_bootstrap.cc:492] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:06.629078 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:06.630440 10082 consensus_queue.cc:1048] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:06.632404 10053 ts_tablet_manager.cc:1434] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.106s user 0.005s sys 0.004s
I20260430 07:58:06.633494 10053 tablet_bootstrap.cc:492] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:06.639164 10060 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:06.650341 10060 raft_consensus.cc:697] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:06.642158 10012 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.654495 10060 consensus_queue.cc:237] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.657455 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:06.658154 9861 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.648208 10218 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.670882 10218 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.682394 10052 tablet_bootstrap.cc:654] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.696087 10012 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:06.698437 9903 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:06.690680 9859 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.699843 10060 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:06.702028 10060 raft_consensus.cc:697] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:06.703775 10060 consensus_queue.cc:237] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.705739 10053 tablet_bootstrap.cc:654] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.712162 9759 catalog_manager.cc:5671] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
W20260430 07:58:06.718866 10244 log.cc:927] Time spent T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.130s user 0.005s sys 0.000s
I20260430 07:58:06.721045 10218 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.727430 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:06.731621 10054 tablet_bootstrap.cc:492] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:06.732182 10054 ts_tablet_manager.cc:1403] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.300s user 0.011s sys 0.032s
I20260430 07:58:06.732064 10219 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:06.700500 9861 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:06.736359 10054 raft_consensus.cc:359] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.737025 10054 raft_consensus.cc:385] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.738205 10054 raft_consensus.cc:740] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.739164 10054 consensus_queue.cc:260] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.740327 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:06.742383 9828 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:06.743247 10113 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:06.744361 10052 tablet_bootstrap.cc:492] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:06.744801 10052 ts_tablet_manager.cc:1403] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.120s user 0.023s sys 0.009s
I20260430 07:58:06.742383 10054 ts_tablet_manager.cc:1434] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260430 07:58:06.751020 10054 tablet_bootstrap.cc:492] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:06.751647 10082 consensus_queue.cc:1048] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:06.767048 10053 tablet_bootstrap.cc:492] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:06.767768 10053 ts_tablet_manager.cc:1403] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.135s user 0.015s sys 0.006s
I20260430 07:58:06.772207 10053 raft_consensus.cc:359] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.773247 10053 raft_consensus.cc:385] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.773648 10053 raft_consensus.cc:740] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.774832 10053 consensus_queue.cc:260] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.778707 10052 raft_consensus.cc:359] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.779518 10052 raft_consensus.cc:385] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.779794 10052 raft_consensus.cc:740] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.780319 10054 tablet_bootstrap.cc:654] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.780553 10052 consensus_queue.cc:260] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.782829 10053 ts_tablet_manager.cc:1434] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.015s user 0.000s sys 0.010s
I20260430 07:58:06.783934 10053 tablet_bootstrap.cc:492] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:06.788394 10052 ts_tablet_manager.cc:1434] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.043s user 0.011s sys 0.001s
I20260430 07:58:06.789793 10052 tablet_bootstrap.cc:492] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:06.790530 10053 tablet_bootstrap.cc:654] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.791775 10054 tablet_bootstrap.cc:492] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:06.792464 10054 ts_tablet_manager.cc:1403] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.006s
I20260430 07:58:06.796085 10054 raft_consensus.cc:359] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.797427 10054 raft_consensus.cc:385] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.797987 10054 raft_consensus.cc:740] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.798928 10054 consensus_queue.cc:260] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.803936 10054 ts_tablet_manager.cc:1434] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260430 07:58:06.804018 10053 tablet_bootstrap.cc:492] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:06.805299 10053 ts_tablet_manager.cc:1403] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.004s
I20260430 07:58:06.805613 10054 tablet_bootstrap.cc:492] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:06.808773 10053 raft_consensus.cc:359] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.809955 10053 raft_consensus.cc:385] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.810386 10053 raft_consensus.cc:740] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.813472 10052 tablet_bootstrap.cc:654] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.813581 10054 tablet_bootstrap.cc:654] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.814311 10053 consensus_queue.cc:260] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.818271 10053 ts_tablet_manager.cc:1434] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
I20260430 07:58:06.820351 10053 tablet_bootstrap.cc:492] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:06.826383 10054 tablet_bootstrap.cc:492] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:06.826922 10054 ts_tablet_manager.cc:1403] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.003s
I20260430 07:58:06.830169 10082 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.829775 10053 tablet_bootstrap.cc:654] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.830649 10082 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.832731 10082 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:06.833829 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:06.834409 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:06.835923 9977 leader_election.cc:343] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.829759 10054 raft_consensus.cc:359] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.838758 10054 raft_consensus.cc:385] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.839179 10054 raft_consensus.cc:740] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.841778 10054 consensus_queue.cc:260] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:06.843429 9978 leader_election.cc:343] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.844197 10054 ts_tablet_manager.cc:1434] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.017s user 0.009s sys 0.000s
I20260430 07:58:06.844224 9978 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:06.846189 10054 tablet_bootstrap.cc:492] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:06.848543 10082 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:06.853715 10054 tablet_bootstrap.cc:654] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.866353 10218 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.867158 10218 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.871634 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:06.872149 10218 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.872486 9937 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:06.875134 10053 tablet_bootstrap.cc:492] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:06.874797 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:06.875839 10053 ts_tablet_manager.cc:1403] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.056s user 0.016s sys 0.032s
I20260430 07:58:06.877969 9828 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:06.879227 9828 leader_election.cc:343] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.880352 10218 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:06.880021 10053 raft_consensus.cc:359] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.880815 10218 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:06.881701 10218 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.883672 10052 tablet_bootstrap.cc:492] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:06.884474 10052 ts_tablet_manager.cc:1403] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.010s
I20260430 07:58:06.889164 10052 raft_consensus.cc:359] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.890224 10052 raft_consensus.cc:385] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.890630 10052 raft_consensus.cc:740] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.891572 10052 consensus_queue.cc:260] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.895812 10052 ts_tablet_manager.cc:1434] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260430 07:58:06.881047 10053 raft_consensus.cc:385] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.896044 10054 tablet_bootstrap.cc:492] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:06.896279 10053 raft_consensus.cc:740] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.896508 10054 ts_tablet_manager.cc:1403] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.051s user 0.024s sys 0.020s
I20260430 07:58:06.890376 10218 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.898571 10052 tablet_bootstrap.cc:492] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:06.899526 10054 raft_consensus.cc:359] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.900862 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:06.901507 9937 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.902184 9761 catalog_manager.cc:5671] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:06.896950 10053 consensus_queue.cc:260] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.904917 9761 catalog_manager.cc:5671] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:06.905740 10148 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.906243 10148 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.907691 10053 ts_tablet_manager.cc:1434] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.031s user 0.011s sys 0.000s
I20260430 07:58:06.908237 10148 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.908653 10053 tablet_bootstrap.cc:492] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:06.910362 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:06.911609 10113 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.912120 10113 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.915364 10113 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.900216 10054 raft_consensus.cc:385] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260430 07:58:06.917171 9903 leader_election.cc:343] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.917557 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:06.919549 10052 tablet_bootstrap.cc:654] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.919296 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:06.922051 9904 leader_election.cc:343] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
W20260430 07:58:06.922578 9828 leader_election.cc:343] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.922503 9904 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:06.923162 9937 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:06.924103 10148 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:06.917752 10054 raft_consensus.cc:740] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.917222 10053 tablet_bootstrap.cc:654] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.925603 10054 consensus_queue.cc:260] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.926795 10218 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.928151 10054 ts_tablet_manager.cc:1434] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20260430 07:58:06.930984 9828 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:06.932644 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:06.933037 10054 tablet_bootstrap.cc:492] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
W20260430 07:58:06.935220 9828 leader_election.cc:343] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.933430 10218 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:06.941833 10218 raft_consensus.cc:697] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:06.942747 10218 consensus_queue.cc:237] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.944628 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:06.951607 10148 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.952355 10148 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.964730 10234 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:06.965322 10234 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.966184 10053 tablet_bootstrap.cc:492] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:06.966184 10148 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.966429 10054 tablet_bootstrap.cc:654] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:06.967289 10053 ts_tablet_manager.cc:1403] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.059s user 0.012s sys 0.013s
I20260430 07:58:06.967330 10234 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:06.970221 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:06.970736 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:06.971133 9861 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:06.971442 10009 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:06.970831 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:06.972455 10012 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
W20260430 07:58:06.972970 9828 leader_election.cc:343] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.973472 9828 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:06.973856 9903 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:06.975931 10113 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:06.976500 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:06.976747 10052 tablet_bootstrap.cc:492] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:06.977443 10052 ts_tablet_manager.cc:1403] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.079s user 0.015s sys 0.005s
I20260430 07:58:06.976861 9903 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:06.981015 10148 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260430 07:58:06.980856 9904 leader_election.cc:343] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:06.981657 10234 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:06.982153 10148 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:06.982185 10234 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:06.982712 10234 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.986251 10148 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:06.987262 10052 raft_consensus.cc:359] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.988077 10052 raft_consensus.cc:385] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.988360 10052 raft_consensus.cc:740] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.989018 10052 consensus_queue.cc:260] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.991350 10052 ts_tablet_manager.cc:1434] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260430 07:58:06.992235 10052 tablet_bootstrap.cc:492] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:06.992362 10053 raft_consensus.cc:359] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:06.993052 10053 raft_consensus.cc:385] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:06.993330 10053 raft_consensus.cc:740] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:06.993860 10053 consensus_queue.cc:260] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.004935 10053 ts_tablet_manager.cc:1434] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.035s user 0.006s sys 0.001s
I20260430 07:58:07.007252 10053 tablet_bootstrap.cc:492] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:07.010464 10054 tablet_bootstrap.cc:492] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:07.084116 10053 tablet_bootstrap.cc:654] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:07.085567 10054 ts_tablet_manager.cc:1403] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.153s user 0.012s sys 0.020s
I20260430 07:58:07.093415 9759 catalog_manager.cc:5671] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:07.094205 10060 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.096124 10060 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.097923 10054 raft_consensus.cc:359] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.102546 10113 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.106223 10054 raft_consensus.cc:385] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.098989 10234 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.106717 10054 raft_consensus.cc:740] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.106750 10113 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.107527 10054 consensus_queue.cc:260] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.109833 10113 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.110932 10060 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.111658 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:07.112610 10234 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.101562 10218 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260430 07:58:07.111752 10052 tablet_bootstrap.cc:654] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:07.115599 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:07.109921 10054 ts_tablet_manager.cc:1434] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260430 07:58:07.116336 9861 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:07.117122 10054 tablet_bootstrap.cc:492] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:07.117583 10231 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.118286 10231 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:07.119278 9904 leader_election.cc:343] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.121358 9904 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:07.122859 10060 raft_consensus.cc:2804] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:07.123299 10060 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:07.128618 10060 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.099056 10148 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.128890 10218 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:07.125962 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:07.130329 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:07.131276 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:07.131608 10148 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.131899 9859 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.132670 10054 tablet_bootstrap.cc:654] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
W20260430 07:58:07.132921 9978 leader_election.cc:343] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.124609 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:07.134334 10231 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:07.136724 10053 tablet_bootstrap.cc:492] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
W20260430 07:58:07.141031 9828 leader_election.cc:343] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.141759 10053 ts_tablet_manager.cc:1403] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.135s user 0.012s sys 0.004s
I20260430 07:58:07.140030 9859 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:07.150491 9904 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:07.151408 10148 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:07.151909 10148 raft_consensus.cc:697] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:07.153069 10148 consensus_queue.cc:237] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.173522 10060 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.181296 9938 raft_consensus.cc:2393] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 1: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:07.184158 10052 tablet_bootstrap.cc:492] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:07.186445 10052 ts_tablet_manager.cc:1403] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.194s user 0.010s sys 0.014s
I20260430 07:58:07.186888 10054 tablet_bootstrap.cc:492] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:07.187827 9828 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:07.192162 10054 ts_tablet_manager.cc:1403] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.075s user 0.014s sys 0.001s
I20260430 07:58:07.192411 10218 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.194118 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:07.194931 10054 raft_consensus.cc:359] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.195626 10054 raft_consensus.cc:385] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.195889 10054 raft_consensus.cc:740] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.196523 10054 consensus_queue.cc:260] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.198197 10111 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.198869 10054 ts_tablet_manager.cc:1434] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260430 07:58:07.199179 10111 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.200449 10054 tablet_bootstrap.cc:492] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:07.201411 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:07.198930 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:07.202841 10007 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.203238 10111 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:07.205848 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:07.208636 9978 leader_election.cc:343] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.210338 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:07.210992 9937 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:07.212342 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:07.214510 10052 raft_consensus.cc:359] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.215598 10052 raft_consensus.cc:385] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.215874 10052 raft_consensus.cc:740] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.215794 9977 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:07.218102 10231 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:07.217741 10053 raft_consensus.cc:359] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.216682 10052 consensus_queue.cc:260] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.218864 10053 raft_consensus.cc:385] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.218614 10231 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:07.220690 10231 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:07.217093 9977 leader_election.cc:343] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.220927 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:07.221901 9977 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:07.222108 10009 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.223318 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:07.208876 10060 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.219419 10053 raft_consensus.cc:740] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.229043 10234 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:07.228911 10053 consensus_queue.cc:260] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.230134 10111 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:07.269011 10113 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.276894 10113 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.317216 10052 ts_tablet_manager.cc:1434] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.110s user 0.027s sys 0.019s
I20260430 07:58:07.289423 10111 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.318425 10052 tablet_bootstrap.cc:492] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:07.226414 9759 catalog_manager.cc:5671] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:07.331595 10054 tablet_bootstrap.cc:654] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
W20260430 07:58:07.232191 9903 leader_election.cc:343] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.334630 10053 ts_tablet_manager.cc:1434] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.192s user 0.018s sys 0.014s
I20260430 07:58:07.336130 10053 tablet_bootstrap.cc:492] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:07.346325 10230 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.346864 10230 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.348798 10230 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:07.350548 10113 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.333629 10234 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260430 07:58:07.354241 10007 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:07.324841 10111 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.358671 10231 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.360638 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:07.361689 10231 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:07.383504 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 54940.8us
W20260430 07:58:07.367578 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 136454us
I20260430 07:58:07.388296 10111 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:07.407497 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:07.412779 10218 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:07.407575 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:07.428472 10009 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
W20260430 07:58:07.430680 9978 leader_election.cc:343] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.431907 10231 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.432627 10231 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.435547 10231 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:07.420964 9903 leader_election.cc:343] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.440052 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:07.463454 9978 leader_election.cc:343] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.474340 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:07.475395 9937 raft_consensus.cc:2393] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 1: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:07.476306 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:07.476958 9936 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.478618 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:07.482077 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:07.481848 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:07.485898 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:07.484745 9977 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:07.489298 10111 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.482304 9903 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:07.493686 10234 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:07.496229 10053 tablet_bootstrap.cc:654] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:07.497973 10234 raft_consensus.cc:697] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:07.498944 10234 consensus_queue.cc:237] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.498003 10010 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:07.502849 9828 leader_election.cc:343] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.503124 10111 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:07.522502 9936 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
W20260430 07:58:07.524320 9828 leader_election.cc:343] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.524827 9828 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:07.526750 10217 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:07.527418 10217 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:07.501152 9861 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:07.531075 9904 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:07.532871 10267 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:07.533614 9977 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:07.535367 9977 leader_election.cc:343] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.535796 9977 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:07.540870 10111 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:07.542552 10231 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:07.536847 9977 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:07.546741 10082 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:07.547008 10232 raft_consensus.cc:2804] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:07.547438 10082 raft_consensus.cc:697] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:07.547616 10232 raft_consensus.cc:697] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:07.548544 10232 consensus_queue.cc:237] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.548565 10082 consensus_queue.cc:237] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.561815 10148 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260430 07:58:07.567739 10219 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260430 07:58:07.579160 10054 tablet_bootstrap.cc:492] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:07.579872 10054 ts_tablet_manager.cc:1403] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.380s user 0.032s sys 0.029s
I20260430 07:58:07.576835 10060 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.605902 10053 tablet_bootstrap.cc:492] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:07.606702 10053 ts_tablet_manager.cc:1403] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.271s user 0.010s sys 0.013s
I20260430 07:58:07.615638 10054 raft_consensus.cc:359] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.616436 10054 raft_consensus.cc:385] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.616705 10054 raft_consensus.cc:740] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.620275 10052 tablet_bootstrap.cc:654] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:07.621641 10054 consensus_queue.cc:260] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.589871 10060 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.626732 10053 raft_consensus.cc:359] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.627313 10053 raft_consensus.cc:385] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.627550 10053 raft_consensus.cc:740] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.628082 10053 consensus_queue.cc:260] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.637779 10054 ts_tablet_manager.cc:1434] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.057s user 0.010s sys 0.001s
I20260430 07:58:07.644977 10060 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.665665 10053 ts_tablet_manager.cc:1434] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.059s user 0.016s sys 0.012s
I20260430 07:58:07.667026 10053 tablet_bootstrap.cc:492] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:07.670608 10054 tablet_bootstrap.cc:492] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:07.665067 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:07.673795 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:07.676789 9904 leader_election.cc:343] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.676270 10005 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:07.685801 10053 tablet_bootstrap.cc:654] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:07.701251 9903 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:07.702706 10234 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:07.703104 10234 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:07.703794 10234 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.705340 10291 consensus_queue.cc:1048] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260430 07:58:07.711723 10060 consensus_queue.cc:1048] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260430 07:58:07.728401 10052 tablet_bootstrap.cc:492] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:07.728955 10052 ts_tablet_manager.cc:1403] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.411s user 0.021s sys 0.015s
W20260430 07:58:07.730626 9809 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:58:07.739557 10234 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.742780 10234 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.744202 10052 raft_consensus.cc:359] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.744932 10052 raft_consensus.cc:385] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.745206 10052 raft_consensus.cc:740] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.748414 10052 consensus_queue.cc:260] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.747141 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:07.756062 9904 leader_election.cc:343] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.757642 10052 ts_tablet_manager.cc:1434] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.028s user 0.011s sys 0.000s
I20260430 07:58:07.759945 10052 tablet_bootstrap.cc:492] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:07.765187 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:07.767338 10010 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.740725 9758 catalog_manager.cc:5671] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
W20260430 07:58:07.775079 10290 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.072s user 0.003s sys 0.003s
I20260430 07:58:07.804904 10052 tablet_bootstrap.cc:654] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:07.808630 9761 catalog_manager.cc:5671] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:07.810565 10289 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:07.811100 10289 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.811348 9761 catalog_manager.cc:5671] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:07.814106 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:07.814945 9860 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:07.816249 9978 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:07.817103 10082 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:07.813210 10289 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:07.817095 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:07.819741 9938 raft_consensus.cc:2393] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:07.817494 10082 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:07.820259 10082 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:07.820016 10010 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:07.814203 10054 tablet_bootstrap.cc:654] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:07.822173 9903 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:07.823412 10053 tablet_bootstrap.cc:492] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:07.823854 10053 ts_tablet_manager.cc:1403] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.157s user 0.032s sys 0.019s
I20260430 07:58:07.825634 10234 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:07.826179 10234 raft_consensus.cc:697] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:07.826977 10234 consensus_queue.cc:237] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.837337 10082 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.867271 10053 raft_consensus.cc:359] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.868029 10053 raft_consensus.cc:385] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.868476 10053 raft_consensus.cc:740] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.869855 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:07.874513 10234 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.874272 10053 consensus_queue.cc:260] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.875005 10234 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.875968 10053 ts_tablet_manager.cc:1434] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.052s user 0.007s sys 0.000s
I20260430 07:58:07.876879 10053 tablet_bootstrap.cc:492] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:07.877950 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:07.878357 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:07.878544 9861 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:07.878916 10010 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:07.880645 9903 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:07.882740 10291 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:07.883158 10291 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:07.883507 10291 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:07.876962 10234 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.886108 10082 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:07.888458 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:07.889798 9978 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:07.890887 10289 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:07.891374 10289 raft_consensus.cc:697] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:07.892220 10289 consensus_queue.cc:237] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.898873 10052 tablet_bootstrap.cc:492] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:07.899369 10052 ts_tablet_manager.cc:1403] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.140s user 0.030s sys 0.019s
I20260430 07:58:07.896852 9761 catalog_manager.cc:5671] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:07.901790 10054 tablet_bootstrap.cc:492] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:07.902387 10054 ts_tablet_manager.cc:1403] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.232s user 0.021s sys 0.028s
I20260430 07:58:07.910861 10054 raft_consensus.cc:359] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.911711 10054 raft_consensus.cc:385] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.911993 10054 raft_consensus.cc:740] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.912694 10054 consensus_queue.cc:260] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.915335 10054 ts_tablet_manager.cc:1434] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.013s user 0.003s sys 0.004s
I20260430 07:58:07.916317 10054 tablet_bootstrap.cc:492] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:07.919534 10291 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.921056 10052 raft_consensus.cc:359] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.921568 9938 raft_consensus.cc:2368] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3.
I20260430 07:58:07.921859 10052 raft_consensus.cc:385] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.922145 10052 raft_consensus.cc:740] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.922775 10052 consensus_queue.cc:260] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.924755 9977 leader_election.cc:400] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3.
I20260430 07:58:07.924935 10052 ts_tablet_manager.cc:1434] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20260430 07:58:07.927520 10053 tablet_bootstrap.cc:654] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:07.930616 10052 tablet_bootstrap.cc:492] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:07.928380 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:07.932557 9861 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:07.935447 10054 tablet_bootstrap.cc:654] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:07.936151 10277 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:07.937114 10277 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.944921 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:07.946513 9938 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:07.947928 10291 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.948261 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:07.949867 9828 leader_election.cc:343] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.951288 10053 tablet_bootstrap.cc:492] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:07.951906 9828 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:07.952509 10053 ts_tablet_manager.cc:1403] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.076s user 0.015s sys 0.001s
I20260430 07:58:07.955034 10217 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:07.955919 10217 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:07.936450 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:07.956791 10277 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.956367 10217 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.960067 10054 tablet_bootstrap.cc:492] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:07.960559 10054 ts_tablet_manager.cc:1403] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.003s
I20260430 07:58:07.936172 9759 catalog_manager.cc:5671] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:07.963977 9861 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:07.964202 10054 raft_consensus.cc:359] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.964911 10054 raft_consensus.cc:385] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.956386 10053 raft_consensus.cc:359] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.965809 10053 raft_consensus.cc:385] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:07.965215 9904 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:07.966128 10053 raft_consensus.cc:740] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.966718 10060 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:07.967379 10060 raft_consensus.cc:697] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:07.967566 10217 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.968197 10060 consensus_queue.cc:237] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.969992 10217 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:07.970487 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:07.971319 9938 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:07.972157 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:07.974324 9828 leader_election.cc:343] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:07.966825 10053 consensus_queue.cc:260] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.979933 10053 ts_tablet_manager.cc:1434] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20260430 07:58:07.979768 9938 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:07.981616 10053 tablet_bootstrap.cc:492] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:07.982535 9828 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:07.984392 10052 tablet_bootstrap.cc:654] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:07.984004 10291 consensus_queue.cc:1048] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:07.965211 10054 raft_consensus.cc:740] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:07.986320 10217 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:07.987067 10217 raft_consensus.cc:697] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:07.986882 10054 consensus_queue.cc:260] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:07.989684 10054 ts_tablet_manager.cc:1434] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.029s user 0.003s sys 0.005s
I20260430 07:58:07.990648 10054 tablet_bootstrap.cc:492] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:08.007033 10053 tablet_bootstrap.cc:654] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.010491 9759 catalog_manager.cc:5671] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 2 to 3, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 3 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:07.987869 10217 consensus_queue.cc:237] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.016523 10054 tablet_bootstrap.cc:654] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.029661 10052 tablet_bootstrap.cc:492] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:08.030215 10052 ts_tablet_manager.cc:1403] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.100s user 0.017s sys 0.009s
W20260430 07:58:08.041054 9904 consensus_peers.cc:597] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 -> Peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Couldn't send request to peer 27f97a330f374849be58a168318c1ec6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:08.049808 10052 raft_consensus.cc:359] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.050570 10052 raft_consensus.cc:385] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.050823 10052 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.050844 10289 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:08.052784 10052 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.057257 10052 ts_tablet_manager.cc:1434] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20260430 07:58:08.060058 10054 tablet_bootstrap.cc:492] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:08.061084 10054 ts_tablet_manager.cc:1403] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.015s
I20260430 07:58:08.063426 10060 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.064095 10060 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.066291 10054 raft_consensus.cc:359] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.067129 10054 raft_consensus.cc:385] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.067404 10054 raft_consensus.cc:740] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.068049 10054 consensus_queue.cc:260] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.068302 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:08.069674 9904 leader_election.cc:343] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.070300 10054 ts_tablet_manager.cc:1434] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260430 07:58:08.070240 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:08.071350 10054 tablet_bootstrap.cc:492] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
W20260430 07:58:08.071564 9903 leader_election.cc:343] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.072021 9903 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:08.073052 10234 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:08.068316 10060 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.075294 10052 tablet_bootstrap.cc:492] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:08.075585 10219 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.076150 10219 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.072290 9759 catalog_manager.cc:5671] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:08.078910 10219 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.080538 10289 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.080737 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:08.081053 10289 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.082998 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:08.083386 10052 tablet_bootstrap.cc:654] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.084009 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:08.084508 9828 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:08.084681 9861 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:08.085845 10219 raft_consensus.cc:2749] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:08.086028 9978 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:08.087581 10082 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:08.087947 10082 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:08.089718 10082 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.091676 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:08.092658 10054 tablet_bootstrap.cc:654] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.088373 10289 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.093746 9860 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.098295 10052 tablet_bootstrap.cc:492] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:08.098767 10052 ts_tablet_manager.cc:1403] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.005s
W20260430 07:58:08.099114 9977 leader_election.cc:343] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.102097 10052 raft_consensus.cc:359] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.103039 10052 raft_consensus.cc:385] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.103333 10052 raft_consensus.cc:740] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.103247 10082 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.108392 10082 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.104058 10052 consensus_queue.cc:260] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.111768 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:08.112452 9859 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.112538 10052 ts_tablet_manager.cc:1434] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.014s user 0.007s sys 0.002s
I20260430 07:58:08.119866 10052 tablet_bootstrap.cc:492] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:08.119997 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:08.121552 10010 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.122063 10054 tablet_bootstrap.cc:492] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:08.123577 10054 ts_tablet_manager.cc:1403] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.003s
I20260430 07:58:08.127024 9859 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
W20260430 07:58:08.127580 9977 leader_election.cc:343] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.127753 10054 raft_consensus.cc:359] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.128554 10054 raft_consensus.cc:385] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.128795 10054 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.129397 10052 tablet_bootstrap.cc:654] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.129750 9978 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:08.131191 10289 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:08.131851 10054 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.132450 10289 raft_consensus.cc:697] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:08.133488 10289 consensus_queue.cc:237] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.134443 10279 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:08.147861 10054 ts_tablet_manager.cc:1434] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260430 07:58:08.148874 10054 tablet_bootstrap.cc:492] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:08.153143 10279 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.153928 10279 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.158036 10279 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.160539 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:08.161352 9859 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:08.161921 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:08.163630 9978 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:08.160205 10234 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260430 07:58:08.164741 9977 leader_election.cc:343] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.166244 10279 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:08.166687 10279 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:08.167042 10279 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.183825 10234 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:08.193995 10054 tablet_bootstrap.cc:654] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.196393 10052 tablet_bootstrap.cc:492] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:08.197371 10052 ts_tablet_manager.cc:1403] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.077s user 0.021s sys 0.028s
I20260430 07:58:08.198132 10082 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.205535 10082 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.201239 10053 tablet_bootstrap.cc:492] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:08.207103 10053 ts_tablet_manager.cc:1403] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.226s user 0.006s sys 0.016s
I20260430 07:58:08.211877 10052 raft_consensus.cc:359] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.212602 10052 raft_consensus.cc:385] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.212877 10052 raft_consensus.cc:740] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.213547 10053 raft_consensus.cc:359] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.222393 10053 raft_consensus.cc:385] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.222961 10053 raft_consensus.cc:740] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.220533 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:08.223701 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:08.224432 9859 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:08.225795 9978 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:08.210853 10279 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.226572 10289 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:08.226989 10289 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:08.227331 10289 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.221925 10082 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.218663 9759 catalog_manager.cc:5671] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:08.224541 9938 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:08.230726 10279 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.232322 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:08.232295 10053 consensus_queue.cc:260] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:08.234009 9977 leader_election.cc:343] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.233201 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:08.235038 10053 ts_tablet_manager.cc:1434] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20260430 07:58:08.235181 9859 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.236671 10053 tablet_bootstrap.cc:492] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:08.240512 10054 tablet_bootstrap.cc:492] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:08.241554 10054 ts_tablet_manager.cc:1403] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.093s user 0.014s sys 0.000s
I20260430 07:58:08.244834 10289 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.247351 10289 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.247807 10054 raft_consensus.cc:359] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.248675 10054 raft_consensus.cc:385] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.249047 10054 raft_consensus.cc:740] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.249621 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:08.246735 10060 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.250285 9938 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.250494 10060 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.249708 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:08.256301 9860 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.216423 10052 consensus_queue.cc:260] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.262231 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:08.263082 10052 ts_tablet_manager.cc:1434] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.065s user 0.008s sys 0.000s
I20260430 07:58:08.260445 9938 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:08.261044 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:08.263943 10009 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:08.264338 10052 tablet_bootstrap.cc:492] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:08.249709 10054 consensus_queue.cc:260] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.263150 9863 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:08.265870 9903 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:08.266652 10234 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:08.267025 10234 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:08.267203 9977 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:08.267458 10234 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.267990 10279 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:08.268472 10279 raft_consensus.cc:697] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:08.269368 10060 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.269335 10279 consensus_queue.cc:237] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.274161 10054 ts_tablet_manager.cc:1434] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.032s user 0.009s sys 0.000s
I20260430 07:58:08.275257 10234 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.261675 10053 tablet_bootstrap.cc:654] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.275599 10054 tablet_bootstrap.cc:492] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:08.276618 10289 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.288149 10289 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.291603 10289 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.291806 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:08.299052 10009 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.289682 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:08.305706 9861 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.305709 9859 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:08.308229 9978 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:08.309862 10279 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:08.292339 10234 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.292752 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:08.310798 9760 catalog_manager.cc:5671] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:08.311909 9938 raft_consensus.cc:2393] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 1: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:08.312249 10279 raft_consensus.cc:697] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:08.313189 10279 consensus_queue.cc:237] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.293547 10054 tablet_bootstrap.cc:654] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.294620 9936 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:08.317577 10234 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:08.320909 10082 consensus_queue.cc:1059] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:08.322235 10082 consensus_queue.cc:1243] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer 77e19ab653f540339c1c613fad1451b5 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260430 07:58:08.322692 10082 raft_consensus.cc:3055] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:08.322971 10082 raft_consensus.cc:740] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:08.323681 10082 consensus_queue.cc:260] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.324726 10082 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:08.295660 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:08.331794 9978 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:08.332686 10231 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.334851 10009 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:08.336871 9903 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:08.337850 10234 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:08.338723 10234 raft_consensus.cc:697] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:08.340363 10234 consensus_queue.cc:237] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.341841 10054 tablet_bootstrap.cc:492] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:08.342389 10054 ts_tablet_manager.cc:1403] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.006s
I20260430 07:58:08.338639 9860 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:08.346109 10054 raft_consensus.cc:359] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.346925 10054 raft_consensus.cc:385] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.347350 10054 raft_consensus.cc:740] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.338017 9861 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:08.351423 10053 tablet_bootstrap.cc:492] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:08.351064 10054 consensus_queue.cc:260] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.351903 10053 ts_tablet_manager.cc:1403] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.116s user 0.020s sys 0.028s
I20260430 07:58:08.353247 10054 ts_tablet_manager.cc:1434] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260430 07:58:08.354295 10054 tablet_bootstrap.cc:492] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:08.355134 10053 raft_consensus.cc:359] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.356353 10053 raft_consensus.cc:385] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.356654 10053 raft_consensus.cc:740] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
W20260430 07:58:08.362567 9904 consensus_peers.cc:597] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 -> Peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Couldn't send request to peer 27f97a330f374849be58a168318c1ec6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:08.362367 10053 consensus_queue.cc:260] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.366503 10053 ts_tablet_manager.cc:1434] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260430 07:58:08.366773 10054 tablet_bootstrap.cc:654] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.367583 10053 tablet_bootstrap.cc:492] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:08.373965 10052 tablet_bootstrap.cc:654] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.370461 9761 catalog_manager.cc:5671] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:08.377120 10231 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:08.383351 10053 tablet_bootstrap.cc:654] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.391485 9859 raft_consensus.cc:1275] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:08.396063 10060 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:08.395391 9758 catalog_manager.cc:5671] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:08.403977 10291 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.404789 10291 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.414252 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:08.414677 10291 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.417225 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:08.421937 9859 raft_consensus.cc:2393] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 1: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:08.423648 9904 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:08.425346 10291 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.433784 10054 tablet_bootstrap.cc:492] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:08.434315 10054 ts_tablet_manager.cc:1403] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.080s user 0.021s sys 0.014s
I20260430 07:58:08.446998 10217 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.448357 10217 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.449818 10277 consensus_queue.cc:1048] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:08.451292 10217 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.454432 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:08.458228 10291 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:08.459276 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:08.461875 10009 raft_consensus.cc:1275] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260430 07:58:08.462268 10009 pending_rounds.cc:85] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 1
I20260430 07:58:08.462595 10009 pending_rounds.cc:107] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260430 07:58:08.462908 10009 raft_consensus.cc:2889] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:08.464093 10052 tablet_bootstrap.cc:492] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:08.464849 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:08.465754 10052 ts_tablet_manager.cc:1403] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.202s user 0.020s sys 0.027s
I20260430 07:58:08.466650 10277 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:08.468003 10232 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:08.471591 10052 raft_consensus.cc:359] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.472254 10052 raft_consensus.cc:385] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.472445 10052 raft_consensus.cc:740] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.473167 10052 consensus_queue.cc:260] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.475714 10052 ts_tablet_manager.cc:1434] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260430 07:58:08.481177 10053 tablet_bootstrap.cc:492] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:08.481714 10053 ts_tablet_manager.cc:1403] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.114s user 0.010s sys 0.014s
I20260430 07:58:08.482141 10054 raft_consensus.cc:359] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.483451 10054 raft_consensus.cc:385] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.483741 10054 raft_consensus.cc:740] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.484560 10054 consensus_queue.cc:260] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.486007 10052 tablet_bootstrap.cc:492] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:08.486461 10053 raft_consensus.cc:359] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.486950 10054 ts_tablet_manager.cc:1434] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.052s user 0.008s sys 0.000s
I20260430 07:58:08.487181 10053 raft_consensus.cc:385] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.487479 10053 raft_consensus.cc:740] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.488200 10054 tablet_bootstrap.cc:492] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:08.488171 10053 consensus_queue.cc:260] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.489986 10053 ts_tablet_manager.cc:1434] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260430 07:58:08.491020 10053 tablet_bootstrap.cc:492] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:08.500424 10053 tablet_bootstrap.cc:654] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.510771 10054 tablet_bootstrap.cc:654] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.513947 9860 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.520342 10279 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:08.524838 10231 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.525367 10231 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.527539 10231 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.529899 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:08.531424 9978 leader_election.cc:343] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.530184 10052 tablet_bootstrap.cc:654] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.535358 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:08.535967 10277 consensus_queue.cc:1048] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260430 07:58:08.538627 9977 leader_election.cc:343] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.539126 9977 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:08.540788 10279 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:08.549901 10279 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.550441 10279 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.553807 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:08.555331 9978 leader_election.cc:343] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.549897 10054 tablet_bootstrap.cc:492] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:08.556802 10054 ts_tablet_manager.cc:1403] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.006s
I20260430 07:58:08.557441 10279 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.557992 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:08.559518 9977 leader_election.cc:343] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.560086 9977 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:08.562752 10231 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:08.563243 10054 raft_consensus.cc:359] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.564359 10054 raft_consensus.cc:385] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.565717 10291 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.566253 10291 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.567602 10053 tablet_bootstrap.cc:492] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:08.564685 10054 raft_consensus.cc:740] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.568267 10053 ts_tablet_manager.cc:1403] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.078s user 0.016s sys 0.002s
I20260430 07:58:08.569345 10291 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.569223 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:08.569005 10054 consensus_queue.cc:260] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:08.570793 9904 leader_election.cc:343] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.574018 10053 raft_consensus.cc:359] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.576450 10053 raft_consensus.cc:385] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.576817 10053 raft_consensus.cc:740] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.575423 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:08.577633 10010 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:08.577613 10053 consensus_queue.cc:260] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.579520 10054 ts_tablet_manager.cc:1434] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20260430 07:58:08.579912 10053 ts_tablet_manager.cc:1434] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20260430 07:58:08.580679 10054 tablet_bootstrap.cc:492] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:08.580817 10053 tablet_bootstrap.cc:492] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:08.581167 10279 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:08.582088 9903 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:08.584338 10060 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:08.584692 10060 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:08.584996 10060 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.607549 10052 tablet_bootstrap.cc:492] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:08.608786 10052 ts_tablet_manager.cc:1403] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.012s
I20260430 07:58:08.610383 10054 tablet_bootstrap.cc:654] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.611632 10060 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.615203 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:08.617395 10053 tablet_bootstrap.cc:654] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.614022 10060 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.619069 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:08.619796 10005 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:08.623420 9904 leader_election.cc:343] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.627599 10052 raft_consensus.cc:359] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.628753 10052 raft_consensus.cc:385] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.629254 10052 raft_consensus.cc:740] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.632200 10052 consensus_queue.cc:260] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.637961 10052 ts_tablet_manager.cc:1434] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.029s user 0.003s sys 0.006s
I20260430 07:58:08.640089 10052 tablet_bootstrap.cc:492] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:08.661506 10005 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:08.662750 9903 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:08.664512 10291 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:08.665086 10291 raft_consensus.cc:697] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:08.667039 10291 consensus_queue.cc:237] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.672870 10053 tablet_bootstrap.cc:492] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:08.673031 10111 consensus_queue.cc:1048] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:08.673406 10053 ts_tablet_manager.cc:1403] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.093s user 0.016s sys 0.000s
I20260430 07:58:08.676177 10053 raft_consensus.cc:359] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.676849 10053 raft_consensus.cc:385] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.677127 10053 raft_consensus.cc:740] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.681819 10052 tablet_bootstrap.cc:654] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.681296 10053 consensus_queue.cc:260] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.686309 10053 ts_tablet_manager.cc:1434] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260430 07:58:08.687609 10053 tablet_bootstrap.cc:492] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:08.687970 10054 tablet_bootstrap.cc:492] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:08.688988 10054 ts_tablet_manager.cc:1403] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.108s user 0.013s sys 0.003s
I20260430 07:58:08.692214 10054 raft_consensus.cc:359] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.693653 10053 tablet_bootstrap.cc:654] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.693212 10054 raft_consensus.cc:385] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.696830 10054 raft_consensus.cc:740] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.697713 10054 consensus_queue.cc:260] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.701103 10052 tablet_bootstrap.cc:492] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:08.701619 10052 ts_tablet_manager.cc:1403] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.008s
I20260430 07:58:08.703136 10054 ts_tablet_manager.cc:1434] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260430 07:58:08.702920 10082 consensus_queue.cc:1048] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:08.704087 10054 tablet_bootstrap.cc:492] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:08.704330 10052 raft_consensus.cc:359] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.704998 10052 raft_consensus.cc:385] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.705251 10052 raft_consensus.cc:740] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.705971 10052 consensus_queue.cc:260] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.708288 10052 ts_tablet_manager.cc:1434] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260430 07:58:08.709856 10052 tablet_bootstrap.cc:492] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:08.727057 10082 consensus_queue.cc:1048] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:08.731297 9760 catalog_manager.cc:5671] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:08.740724 10054 tablet_bootstrap.cc:654] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.742872 10060 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.743301 10060 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.745102 10060 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.750334 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:08.751735 9904 leader_election.cc:343] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.751806 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:08.752485 10010 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:08.762364 9861 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.764858 9903 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:08.769178 10060 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:08.769613 10060 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:08.769935 10060 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.783267 10052 tablet_bootstrap.cc:654] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.791405 10060 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.792116 10053 tablet_bootstrap.cc:492] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:08.792763 10053 ts_tablet_manager.cc:1403] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.105s user 0.025s sys 0.017s
I20260430 07:58:08.793706 10060 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.805804 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:08.806820 10010 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.808794 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:08.810631 9904 leader_election.cc:343] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.823606 10010 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:08.826295 9903 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:08.826987 10053 raft_consensus.cc:359] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.827853 10053 raft_consensus.cc:385] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.828202 10053 raft_consensus.cc:740] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.829008 9938 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.829089 10053 consensus_queue.cc:260] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.835954 10234 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:08.836477 10234 raft_consensus.cc:697] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:08.837767 10234 consensus_queue.cc:237] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.839279 10053 ts_tablet_manager.cc:1434] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.046s user 0.005s sys 0.005s
I20260430 07:58:08.840556 10053 tablet_bootstrap.cc:492] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:08.852751 10052 tablet_bootstrap.cc:492] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:08.853291 10052 ts_tablet_manager.cc:1403] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.144s user 0.012s sys 0.021s
I20260430 07:58:08.855806 10052 raft_consensus.cc:359] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.856343 10052 raft_consensus.cc:385] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.856508 10052 raft_consensus.cc:740] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.857003 10052 consensus_queue.cc:260] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.858944 10052 ts_tablet_manager.cc:1434] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260430 07:58:08.860049 10052 tablet_bootstrap.cc:492] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:08.867100 10052 tablet_bootstrap.cc:654] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.867707 10053 tablet_bootstrap.cc:654] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.869880 9761 catalog_manager.cc:5671] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:08.875945 10060 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.876494 10060 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.886775 10111 consensus_queue.cc:1048] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260430 07:58:08.888185 9809 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:58:08.887167 10053 tablet_bootstrap.cc:492] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:08.889389 10053 ts_tablet_manager.cc:1403] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.014s
I20260430 07:58:08.893111 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:08.887633 10052 tablet_bootstrap.cc:492] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:08.894178 10052 ts_tablet_manager.cc:1403] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.004s
I20260430 07:58:08.895351 10053 raft_consensus.cc:359] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.896435 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:08.896579 10053 raft_consensus.cc:385] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.897200 10053 raft_consensus.cc:740] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.890300 10054 tablet_bootstrap.cc:492] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:08.898125 9904 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:08.891952 10060 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.898846 10054 ts_tablet_manager.cc:1403] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.195s user 0.017s sys 0.007s
I20260430 07:58:08.898991 10052 raft_consensus.cc:359] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.899752 10052 raft_consensus.cc:385] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.901248 10052 raft_consensus.cc:740] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.901885 10054 raft_consensus.cc:359] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.904084 10054 raft_consensus.cc:385] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:08.903826 10052 consensus_queue.cc:260] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.904597 10054 raft_consensus.cc:740] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:08.905892 10054 consensus_queue.cc:260] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.906513 10052 ts_tablet_manager.cc:1434] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.012s user 0.005s sys 0.004s
I20260430 07:58:08.908069 10054 ts_tablet_manager.cc:1434] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260430 07:58:08.897921 10053 consensus_queue.cc:260] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.910017 10054 tablet_bootstrap.cc:492] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:08.910066 10053 ts_tablet_manager.cc:1434] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260430 07:58:08.911080 10053 tablet_bootstrap.cc:492] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:08.918848 10053 tablet_bootstrap.cc:654] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.925477 10234 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:08.932722 10111 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.933239 10111 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.934940 10052 tablet_bootstrap.cc:492] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:08.936292 10111 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.937456 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:08.941377 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:08.943830 9938 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
W20260430 07:58:08.945916 9978 leader_election.cc:343] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.947316 9977 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:08.948167 10111 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:08.948539 10111 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:08.948869 10111 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.950881 10234 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:08.951642 10082 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.952085 10082 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.953907 10082 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:08.956377 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:08.956775 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:08.957104 9938 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
W20260430 07:58:08.960068 9978 leader_election.cc:343] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.960536 9978 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:08.961021 10337 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.962627 10082 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:08.963135 10082 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:08.963454 10082 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.961694 10337 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.967058 10337 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.967751 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:08.967867 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:08.968441 9937 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:08.968894 10010 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:08.969848 10267 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:08.970223 9828 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:08.970552 10217 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:08.970956 10277 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:08.971346 10277 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:08.971623 10277 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.977653 10053 tablet_bootstrap.cc:492] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:08.978209 10053 ts_tablet_manager.cc:1403] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.067s user 0.015s sys 0.003s
I20260430 07:58:08.971048 10217 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.984014 10082 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.985605 10217 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:08.986667 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:08.989108 9828 leader_election.cc:343] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:08.986644 10111 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:08.987424 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:08.990672 10010 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:08.992131 9828 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:08.993036 10052 tablet_bootstrap.cc:654] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.993194 10217 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:08.993677 10217 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:08.994032 10217 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:08.999701 10054 tablet_bootstrap.cc:654] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:08.993038 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:09.010332 9978 leader_election.cc:343] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:09.017565 10082 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:09.021044 10111 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:09.024627 10053 raft_consensus.cc:359] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.027688 10053 raft_consensus.cc:385] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:09.028101 10053 raft_consensus.cc:740] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:09.029003 10053 consensus_queue.cc:260] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.034534 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:09.044797 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:09.045492 9937 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:09.050673 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:09.051332 9938 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:09.052634 10217 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:09.053779 9978 leader_election.cc:343] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:09.055019 10217 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:09.055308 10053 ts_tablet_manager.cc:1434] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.077s user 0.008s sys 0.000s
I20260430 07:58:09.056445 10053 tablet_bootstrap.cc:492] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:09.094071 9937 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:09.076812 10277 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.101742 9977 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:09.103006 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:09.103760 10009 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:09.118427 10277 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:09.119267 10111 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:09.131445 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:09.138384 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:09.140791 9934 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:09.140611 10012 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:09.129698 10111 raft_consensus.cc:697] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:09.147017 10111 consensus_queue.cc:237] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.148537 10053 tablet_bootstrap.cc:654] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:09.159190 10219 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:09.159776 10219 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.129448 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:09.172233 9828 leader_election.cc:343] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:09.176421 10054 tablet_bootstrap.cc:492] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:09.177125 10054 ts_tablet_manager.cc:1403] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.267s user 0.044s sys 0.046s
I20260430 07:58:09.178570 10219 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:09.181632 10052 tablet_bootstrap.cc:492] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:09.182197 10052 ts_tablet_manager.cc:1403] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.254s user 0.013s sys 0.015s
I20260430 07:58:09.174757 10012 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:09.199031 10052 raft_consensus.cc:359] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.200119 10052 raft_consensus.cc:385] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:09.200503 10052 raft_consensus.cc:740] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:09.203356 9828 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:09.205046 10052 consensus_queue.cc:260] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.199680 10009 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:09.204707 10358 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:09.206503 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:09.207739 10358 raft_consensus.cc:697] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:09.213119 10052 ts_tablet_manager.cc:1434] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.031s user 0.010s sys 0.000s
I20260430 07:58:09.214607 10052 tablet_bootstrap.cc:492] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:09.208585 10358 consensus_queue.cc:237] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.219143 9828 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:09.206043 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:09.222687 10277 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:09.223713 10005 raft_consensus.cc:2393] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 1: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:09.224459 10277 raft_consensus.cc:697] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:09.225108 9934 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:09.226069 9828 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:09.229664 10219 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:09.233649 10337 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:09.233807 9938 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:09.234237 10337 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.236816 9977 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:09.246822 10277 consensus_queue.cc:237] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.247437 10082 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:09.247989 10082 raft_consensus.cc:697] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:09.248706 10082 consensus_queue.cc:237] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.249305 10337 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:09.278213 10054 raft_consensus.cc:359] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.279201 10054 raft_consensus.cc:385] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:09.279505 10054 raft_consensus.cc:740] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:09.280155 10054 consensus_queue.cc:260] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.285035 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:09.310168 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:09.336196 10052 tablet_bootstrap.cc:654] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:09.326144 10053 tablet_bootstrap.cc:492] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
W20260430 07:58:09.346547 9828 leader_election.cc:343] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:09.347426 10060 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:09.347437 10053 ts_tablet_manager.cc:1403] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.291s user 0.022s sys 0.017s
I20260430 07:58:09.353464 10054 ts_tablet_manager.cc:1434] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.176s user 0.008s sys 0.018s
W20260430 07:58:09.354987 9828 leader_election.cc:343] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:09.355451 9828 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:09.350886 10219 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:09.357204 10358 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:09.368389 10054 tablet_bootstrap.cc:492] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:09.390703 10060 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260430 07:58:09.377897 10361 log.cc:927] Time spent T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.116s user 0.001s sys 0.000s
I20260430 07:58:09.379122 9758 catalog_manager.cc:5671] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:09.414976 10053 raft_consensus.cc:359] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.415748 10053 raft_consensus.cc:385] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:09.416397 10053 raft_consensus.cc:740] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:09.402102 10232 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:09.422608 10232 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.425997 10232 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:09.424157 10277 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:09.417430 10053 consensus_queue.cc:260] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.429562 10277 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.436261 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:09.437791 10006 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:09.439569 10277 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:09.443550 10053 ts_tablet_manager.cc:1434] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.093s user 0.009s sys 0.000s
I20260430 07:58:09.446169 10052 tablet_bootstrap.cc:492] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:09.447497 10053 tablet_bootstrap.cc:492] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
W20260430 07:58:09.502509 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 74205.8us
I20260430 07:58:09.511510 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:09.514946 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 67913.9us
I20260430 07:58:09.542876 10052 ts_tablet_manager.cc:1403] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.329s user 0.034s sys 0.040s
I20260430 07:58:09.556102 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:09.559283 10234 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:09.571633 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:09.586084 10054 tablet_bootstrap.cc:654] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:09.560618 10234 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.648638 9760 catalog_manager.cc:5671] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:09.766345 10234 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:09.779386 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 93319.8us
W20260430 07:58:09.780712 9904 leader_election.cc:343] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:09.798837 10053 tablet_bootstrap.cc:654] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
W20260430 07:58:09.800701 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 177007us
W20260430 07:58:09.808326 9904 consensus_peers.cc:597] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 -> Peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Couldn't send request to peer 27f97a330f374849be58a168318c1ec6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:09.816035 10052 raft_consensus.cc:359] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.821580 10052 raft_consensus.cc:385] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:09.822026 10052 raft_consensus.cc:740] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:09.822734 10052 consensus_queue.cc:260] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:09.833592 9828 leader_election.cc:343] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:09.842348 10052 ts_tablet_manager.cc:1434] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.297s user 0.017s sys 0.017s
I20260430 07:58:09.856470 10052 tablet_bootstrap.cc:492] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:09.874915 10376 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:09.875716 10376 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.891358 10376 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:09.900569 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:09.907456 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:09.909377 10054 tablet_bootstrap.cc:492] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
W20260430 07:58:09.917330 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 82571.8us
I20260430 07:58:09.917522 9903 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:09.923950 10375 consensus_queue.cc:1048] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260430 07:58:09.934778 10376 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:09.935783 10376 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:09.936530 10376 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:09.948710 10054 ts_tablet_manager.cc:1403] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.581s user 0.012s sys 0.019s
I20260430 07:58:09.956908 10054 raft_consensus.cc:359] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.962869 10054 raft_consensus.cc:385] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:09.963747 10054 raft_consensus.cc:740] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:09.972777 10054 consensus_queue.cc:260] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:09.977780 9828 leader_election.cc:343] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:09.978247 9828 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:09.978281 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:09.972692 9903 leader_election.cc:343] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:09.978583 10054 ts_tablet_manager.cc:1434] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.029s user 0.004s sys 0.004s
I20260430 07:58:09.982854 10054 tablet_bootstrap.cc:492] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:09.976630 10052 tablet_bootstrap.cc:654] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:09.988224 10385 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:09.998600 10385 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:09.999658 10385 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:09.999950 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:10.002168 9828 consensus_peers.cc:597] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 -> Peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Couldn't send request to peer 77e19ab653f540339c1c613fad1451b5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:10.017505 10385 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:10.017405 10367 consensus_queue.cc:1048] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260430 07:58:10.029254 9859 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:10.032073 9903 leader_election.cc:343] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.046489 9760 catalog_manager.cc:5671] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:10.050751 10376 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:10.059458 9904 leader_election.cc:343] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.060523 9904 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:10.065692 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:10.068150 10052 tablet_bootstrap.cc:492] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:10.070662 10052 ts_tablet_manager.cc:1403] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.215s user 0.024s sys 0.025s
I20260430 07:58:10.053164 10376 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:10.078603 10277 consensus_queue.cc:1048] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260430 07:58:10.079571 10217 consensus_queue.cc:1048] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260430 07:58:10.081598 9978 consensus_peers.cc:597] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 -> Peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Couldn't send request to peer 27f97a330f374849be58a168318c1ec6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:10.091614 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:10.092348 10008 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:10.093724 10052 raft_consensus.cc:359] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.094760 10052 raft_consensus.cc:385] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.095055 10052 raft_consensus.cc:740] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.096006 10052 consensus_queue.cc:260] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.100520 10053 tablet_bootstrap.cc:492] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:10.104128 10053 ts_tablet_manager.cc:1403] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.657s user 0.036s sys 0.035s
I20260430 07:58:10.100358 10358 consensus_queue.cc:1048] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260430 07:58:10.106221 10052 ts_tablet_manager.cc:1434] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.035s user 0.010s sys 0.002s
I20260430 07:58:10.119196 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:10.124348 10377 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:10.129731 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:10.134829 10052 tablet_bootstrap.cc:492] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:10.138718 9828 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:10.139523 9758 catalog_manager.cc:5671] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:10.142050 10054 tablet_bootstrap.cc:654] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.149979 9863 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:10.148128 10053 raft_consensus.cc:359] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.154572 10053 raft_consensus.cc:385] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.156172 10053 raft_consensus.cc:740] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.166399 10053 consensus_queue.cc:260] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:10.192575 9904 leader_election.cc:343] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.193159 9904 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:10.204949 10053 ts_tablet_manager.cc:1434] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.100s user 0.009s sys 0.004s
I20260430 07:58:10.209973 10053 tablet_bootstrap.cc:492] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:10.210665 10232 raft_consensus.cc:2749] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260430 07:58:10.209448 9904 leader_election.cc:343] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.196593 10367 consensus_queue.cc:1048] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20260430 07:58:10.252262 10381 consensus_queue.cc:1048] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:10.271677 10053 tablet_bootstrap.cc:654] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.272607 10052 tablet_bootstrap.cc:654] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.307224 10054 tablet_bootstrap.cc:492] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:10.307718 10054 ts_tablet_manager.cc:1403] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.326s user 0.022s sys 0.034s
I20260430 07:58:10.310490 10054 raft_consensus.cc:359] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.311228 10054 raft_consensus.cc:385] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.311508 10054 raft_consensus.cc:740] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.312156 10054 consensus_queue.cc:260] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.351758 10082 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:10.351876 10008 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:10.352294 10082 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.355348 10082 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:10.379869 10358 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:10.386058 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:10.405351 10054 ts_tablet_manager.cc:1434] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.097s user 0.008s sys 0.001s
I20260430 07:58:10.406831 10054 tablet_bootstrap.cc:492] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
W20260430 07:58:10.413815 9977 leader_election.cc:343] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.412148 9903 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:10.444298 10053 tablet_bootstrap.cc:492] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:10.445329 10053 ts_tablet_manager.cc:1403] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.236s user 0.010s sys 0.017s
I20260430 07:58:10.445683 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:10.447384 10052 tablet_bootstrap.cc:492] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:10.447994 10052 ts_tablet_manager.cc:1403] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.314s user 0.012s sys 0.013s
I20260430 07:58:10.466041 9863 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:10.480475 10052 raft_consensus.cc:359] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.481364 10052 raft_consensus.cc:385] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.481670 10052 raft_consensus.cc:740] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.482440 10054 tablet_bootstrap.cc:654] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.482349 10052 consensus_queue.cc:260] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.514938 10373 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:10.508107 10379 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:10.519845 10379 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:10.518688 9809 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:58:10.533422 10373 raft_consensus.cc:697] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:10.534498 10373 consensus_queue.cc:237] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.536226 10379 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:10.547724 9978 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:10.550416 10111 raft_consensus.cc:2804] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:10.550832 10111 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:10.551155 10111 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:10.553202 10052 ts_tablet_manager.cc:1434] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.105s user 0.008s sys 0.008s
I20260430 07:58:10.555867 10053 raft_consensus.cc:359] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.557552 10053 raft_consensus.cc:385] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.557860 10053 raft_consensus.cc:740] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.567507 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:10.580749 9903 leader_election.cc:343] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.592458 10053 consensus_queue.cc:260] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.597988 10052 tablet_bootstrap.cc:492] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:10.617988 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:10.618714 9860 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:10.618714 10053 ts_tablet_manager.cc:1434] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.173s user 0.012s sys 0.000s
I20260430 07:58:10.628793 9904 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:10.639385 10054 tablet_bootstrap.cc:492] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:10.639983 10054 ts_tablet_manager.cc:1403] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.233s user 0.019s sys 0.026s
I20260430 07:58:10.645426 10373 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:10.645881 10373 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:10.646270 10373 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:10.649989 10053 tablet_bootstrap.cc:492] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:10.662400 10111 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.667527 9758 catalog_manager.cc:5671] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:10.673029 10053 tablet_bootstrap.cc:654] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.674484 10378 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:10.675210 10378 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.677737 10378 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:10.682783 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:10.692379 10373 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.703120 10054 raft_consensus.cc:359] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.703869 10054 raft_consensus.cc:385] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.704138 10054 raft_consensus.cc:740] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.704808 10054 consensus_queue.cc:260] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.706276 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:10.707248 10054 ts_tablet_manager.cc:1434] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.067s user 0.015s sys 0.023s
W20260430 07:58:10.708441 9903 leader_election.cc:343] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.711448 10054 tablet_bootstrap.cc:492] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
W20260430 07:58:10.711550 9903 leader_election.cc:343] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.710893 10373 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:10.713780 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:10.714565 9860 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:10.715137 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:10.716574 10111 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:10.718947 9904 leader_election.cc:343] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.720386 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:10.730710 9860 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:10.719998 9904 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:10.734161 10379 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260430 07:58:10.741139 9977 leader_election.cc:343] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.733605 9904 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:10.743945 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:10.744724 9859 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:10.743453 10378 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:10.751482 10378 raft_consensus.cc:697] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:10.761405 10378 consensus_queue.cc:237] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.762387 10230 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:10.763432 10230 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.768493 10230 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:10.772995 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:10.773926 9859 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:10.774403 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:10.774889 9977 leader_election.cc:343] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
W20260430 07:58:10.775907 9978 leader_election.cc:343] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.776415 9978 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:10.777330 10111 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:10.780380 9978 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:10.780965 10054 tablet_bootstrap.cc:654] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.782583 10230 raft_consensus.cc:2804] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:10.783139 10230 raft_consensus.cc:697] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:10.784531 10052 tablet_bootstrap.cc:654] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.786062 10230 consensus_queue.cc:237] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.794833 10053 tablet_bootstrap.cc:492] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:10.795639 10053 ts_tablet_manager.cc:1403] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.146s user 0.017s sys 0.019s
I20260430 07:58:10.805002 10053 raft_consensus.cc:359] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.805797 10053 raft_consensus.cc:385] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.806298 9760 catalog_manager.cc:5671] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:10.806100 10053 raft_consensus.cc:740] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.809988 10053 consensus_queue.cc:260] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.812541 10053 ts_tablet_manager.cc:1434] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260430 07:58:10.813700 10053 tablet_bootstrap.cc:492] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:10.821604 10054 tablet_bootstrap.cc:492] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:10.822741 10054 ts_tablet_manager.cc:1403] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.115s user 0.009s sys 0.007s
I20260430 07:58:10.846477 10054 raft_consensus.cc:359] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.847227 10054 raft_consensus.cc:385] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.847520 10054 raft_consensus.cc:740] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.848177 10054 consensus_queue.cc:260] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.850678 10054 ts_tablet_manager.cc:1434] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.027s user 0.004s sys 0.002s
I20260430 07:58:10.850549 10082 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:10.851780 10054 tablet_bootstrap.cc:492] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:10.852007 10082 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.853897 10082 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:10.855916 10053 tablet_bootstrap.cc:654] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.856535 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:10.858901 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:10.859398 9978 leader_election.cc:343] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
W20260430 07:58:10.860908 9977 leader_election.cc:343] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:10.864173 9977 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:10.865650 10111 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:10.849330 9761 catalog_manager.cc:5671] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:10.868556 10054 tablet_bootstrap.cc:654] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.875371 10052 tablet_bootstrap.cc:492] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:10.875885 10052 ts_tablet_manager.cc:1403] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.288s user 0.024s sys 0.036s
I20260430 07:58:10.876055 10053 tablet_bootstrap.cc:492] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:10.876834 10053 ts_tablet_manager.cc:1403] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.063s user 0.015s sys 0.007s
I20260430 07:58:10.879731 10052 raft_consensus.cc:359] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.880900 10052 raft_consensus.cc:385] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.881301 10052 raft_consensus.cc:740] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.881841 10054 tablet_bootstrap.cc:492] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:10.881464 10053 raft_consensus.cc:359] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.882458 10053 raft_consensus.cc:385] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.882532 10054 ts_tablet_manager.cc:1403] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.012s
I20260430 07:58:10.883291 10053 raft_consensus.cc:740] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.884227 10053 consensus_queue.cc:260] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.882511 10052 consensus_queue.cc:260] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.886693 10054 raft_consensus.cc:359] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.887043 10053 ts_tablet_manager.cc:1434] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20260430 07:58:10.887470 10054 raft_consensus.cc:385] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.887796 10054 raft_consensus.cc:740] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.887017 10052 ts_tablet_manager.cc:1434] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260430 07:58:10.889088 10053 tablet_bootstrap.cc:492] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:10.888757 10054 consensus_queue.cc:260] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.890650 10052 tablet_bootstrap.cc:492] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:10.896637 10054 ts_tablet_manager.cc:1434] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260430 07:58:10.898622 10054 tablet_bootstrap.cc:492] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:10.900549 10053 tablet_bootstrap.cc:654] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.910239 10052 tablet_bootstrap.cc:654] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.917420 10054 tablet_bootstrap.cc:654] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.925371 10052 tablet_bootstrap.cc:492] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:10.926194 10052 ts_tablet_manager.cc:1403] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.004s
I20260430 07:58:10.931635 10052 raft_consensus.cc:359] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.932338 10052 raft_consensus.cc:385] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:10.932590 10052 raft_consensus.cc:740] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:10.933329 10052 consensus_queue.cc:260] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:10.935303 10052 ts_tablet_manager.cc:1434] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260430 07:58:10.936618 10052 tablet_bootstrap.cc:492] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:10.983558 10052 tablet_bootstrap.cc:654] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:10.983544 9860 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:10.996270 10054 tablet_bootstrap.cc:492] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:10.996798 10054 ts_tablet_manager.cc:1403] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.098s user 0.017s sys 0.023s
I20260430 07:58:10.996721 10379 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:10.997813 10053 tablet_bootstrap.cc:492] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:10.998363 10053 ts_tablet_manager.cc:1403] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.110s user 0.005s sys 0.048s
I20260430 07:58:11.001045 10054 raft_consensus.cc:359] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.001971 10054 raft_consensus.cc:385] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.002245 10054 raft_consensus.cc:740] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.003453 10054 consensus_queue.cc:260] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.002485 10053 raft_consensus.cc:359] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.005374 10053 raft_consensus.cc:385] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.005739 10053 raft_consensus.cc:740] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.005703 10054 ts_tablet_manager.cc:1434] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260430 07:58:11.006938 10379 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:11.006814 10053 consensus_queue.cc:260] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.008551 10054 tablet_bootstrap.cc:492] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:11.014014 10053 ts_tablet_manager.cc:1434] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260430 07:58:11.015836 10053 tablet_bootstrap.cc:492] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:11.022655 10052 tablet_bootstrap.cc:492] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:11.024222 10052 ts_tablet_manager.cc:1403] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.013s
I20260430 07:58:11.041604 10052 raft_consensus.cc:359] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.042321 10052 raft_consensus.cc:385] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.042588 10052 raft_consensus.cc:740] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.044962 10052 consensus_queue.cc:260] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.052877 10054 tablet_bootstrap.cc:654] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.053529 10052 ts_tablet_manager.cc:1434] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.029s user 0.009s sys 0.000s
I20260430 07:58:11.054523 10052 tablet_bootstrap.cc:492] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:11.055929 10053 tablet_bootstrap.cc:654] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.058738 10379 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.059470 10379 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.061331 10415 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.061836 10415 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.062136 10379 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.063886 10415 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.064425 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:11.066836 9904 leader_election.cc:343] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.066751 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:11.067589 10008 raft_consensus.cc:2468] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:11.070663 9903 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:11.071897 10379 raft_consensus.cc:2804] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:11.072268 10379 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:11.072582 10379 raft_consensus.cc:3060] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.073769 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:11.078935 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:11.080615 10053 tablet_bootstrap.cc:492] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:11.081578 10053 ts_tablet_manager.cc:1403] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.013s
I20260430 07:58:11.082348 9828 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:11.084235 10415 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:11.079202 10052 tablet_bootstrap.cc:654] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.084474 10053 raft_consensus.cc:359] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.092525 10053 raft_consensus.cc:385] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.095216 10379 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.096889 10053 raft_consensus.cc:740] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.098380 10053 consensus_queue.cc:260] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.099368 10379 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.099493 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:11.100544 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:11.102655 10010 raft_consensus.cc:3060] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:11.102095 9904 leader_election.cc:343] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.102663 10053 ts_tablet_manager.cc:1434] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260430 07:58:11.106709 10053 tablet_bootstrap.cc:492] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:11.106757 10052 tablet_bootstrap.cc:492] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:11.107432 10052 ts_tablet_manager.cc:1403] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.004s
I20260430 07:58:11.110427 10052 raft_consensus.cc:359] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.111614 10052 raft_consensus.cc:385] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.111994 10052 raft_consensus.cc:740] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.112756 10052 consensus_queue.cc:260] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.113696 10053 tablet_bootstrap.cc:654] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.117316 10052 ts_tablet_manager.cc:1434] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260430 07:58:11.121500 10052 tablet_bootstrap.cc:492] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:11.124429 10230 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.125077 10230 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.128695 10052 tablet_bootstrap.cc:654] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.129247 10010 raft_consensus.cc:2468] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:11.129107 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:11.131570 9978 leader_election.cc:343] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.133524 9903 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:11.134460 10379 raft_consensus.cc:2804] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:11.135188 10379 raft_consensus.cc:697] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:11.136384 10379 consensus_queue.cc:237] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.141151 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:11.142814 9977 leader_election.cc:343] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.143364 9977 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:11.144332 10111 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:11.139600 10230 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:11.148204 10054 tablet_bootstrap.cc:492] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:11.148685 10054 ts_tablet_manager.cc:1403] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.140s user 0.004s sys 0.019s
I20260430 07:58:11.150494 10277 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.151175 10277 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.151348 10054 raft_consensus.cc:359] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.152348 10054 raft_consensus.cc:385] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.152657 10054 raft_consensus.cc:740] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.153331 10054 consensus_queue.cc:260] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.156592 10054 ts_tablet_manager.cc:1434] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260430 07:58:11.156713 10277 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.158164 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:11.158936 10052 tablet_bootstrap.cc:492] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:11.159121 10054 tablet_bootstrap.cc:492] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
W20260430 07:58:11.159399 9828 leader_election.cc:343] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.159552 10052 ts_tablet_manager.cc:1403] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.039s user 0.019s sys 0.010s
I20260430 07:58:11.163874 10052 raft_consensus.cc:359] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.165624 10052 raft_consensus.cc:385] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.166669 10052 raft_consensus.cc:740] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.167378 10006 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.167698 10054 tablet_bootstrap.cc:654] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.175402 9758 catalog_manager.cc:5671] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:11.167472 10052 consensus_queue.cc:260] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.175860 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:11.177204 9936 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:11.178206 9933 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.181762 9828 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:11.182351 10232 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:11.183210 10052 ts_tablet_manager.cc:1434] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.023s user 0.009s sys 0.000s
I20260430 07:58:11.184826 10052 tablet_bootstrap.cc:492] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:11.187467 10277 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:11.187909 10277 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:11.188252 10277 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.188661 10373 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.189306 10373 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.192010 10373 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.193588 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:11.194453 9862 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:11.196038 9904 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:11.196142 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:11.196760 10008 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:11.197017 10232 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:11.197485 10232 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:11.200883 10232 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.201218 10053 tablet_bootstrap.cc:492] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:11.201747 10053 ts_tablet_manager.cc:1403] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.095s user 0.004s sys 0.013s
I20260430 07:58:11.205670 10052 tablet_bootstrap.cc:654] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.204933 10053 raft_consensus.cc:359] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.208904 10053 raft_consensus.cc:385] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.209218 10053 raft_consensus.cc:740] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.209868 10053 consensus_queue.cc:260] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.212201 10053 ts_tablet_manager.cc:1434] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.010s user 0.001s sys 0.003s
I20260430 07:58:11.225328 10053 tablet_bootstrap.cc:492] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:11.228369 10232 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.231362 9862 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.234047 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:11.234655 10053 tablet_bootstrap.cc:654] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.234659 9860 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.246449 9860 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:11.231380 10277 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.248416 9904 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:11.247644 10232 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.249485 10378 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:11.249991 10378 raft_consensus.cc:697] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:11.250901 10378 consensus_queue.cc:237] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.256415 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:11.257633 9936 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.271427 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:11.273188 9828 leader_election.cc:343] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.275168 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:11.277015 10009 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.341971 10277 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.346700 10053 tablet_bootstrap.cc:492] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:11.347245 10053 ts_tablet_manager.cc:1403] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.134s user 0.035s sys 0.022s
W20260430 07:58:11.350427 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 72281.2us
I20260430 07:58:11.350311 10053 raft_consensus.cc:359] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.351085 10053 raft_consensus.cc:385] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.351385 10053 raft_consensus.cc:740] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.352008 10053 consensus_queue.cc:260] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.353150 10375 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:11.353689 10375 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.358052 10053 ts_tablet_manager.cc:1434] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260430 07:58:11.359217 10053 tablet_bootstrap.cc:492] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:11.366418 10375 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.370756 10375 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:11.372650 9937 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.379390 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:11.381017 10052 tablet_bootstrap.cc:492] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:11.381572 10052 ts_tablet_manager.cc:1403] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.197s user 0.030s sys 0.053s
I20260430 07:58:11.398437 10054 tablet_bootstrap.cc:492] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:11.399066 10054 ts_tablet_manager.cc:1403] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.240s user 0.014s sys 0.007s
I20260430 07:58:11.411650 10052 raft_consensus.cc:359] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.413821 10052 raft_consensus.cc:385] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.414086 10052 raft_consensus.cc:740] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.414638 10052 consensus_queue.cc:260] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.433209 10052 ts_tablet_manager.cc:1434] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.051s user 0.005s sys 0.002s
I20260430 07:58:11.434381 10052 tablet_bootstrap.cc:492] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:11.441825 10053 tablet_bootstrap.cc:654] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.442883 10082 consensus_queue.cc:1048] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:11.447407 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:11.429912 10415 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.449904 10415 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.452117 10415 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.454533 10009 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:11.454902 9936 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:11.448325 10082 consensus_queue.cc:1048] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
W20260430 07:58:11.472431 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 49033us
I20260430 07:58:11.475946 9828 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:11.473538 10111 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.480655 10409 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:11.476879 10111 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.481411 10409 raft_consensus.cc:697] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:11.482966 10409 consensus_queue.cc:237] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.483700 10054 raft_consensus.cc:359] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.484650 10054 raft_consensus.cc:385] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.484937 10054 raft_consensus.cc:740] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.485643 10054 consensus_queue.cc:260] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.486441 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:11.484081 10111 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:11.487217 9861 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:11.490483 9978 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:11.491266 10111 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:11.491681 10111 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:11.491989 10111 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.503593 10217 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.503360 10378 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.508000 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:11.508154 10217 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.508332 10378 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.509533 9936 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:11.510614 10217 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.510660 10378 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.527683 9903 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:11.527839 9828 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:11.493808 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:11.534092 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:11.610584 10111 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.636214 9934 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:11.547027 10217 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:11.570156 10054 ts_tablet_manager.cc:1434] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.171s user 0.005s sys 0.005s
I20260430 07:58:11.641727 10217 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:11.642230 10217 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.642758 10054 tablet_bootstrap.cc:492] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:11.642696 10053 tablet_bootstrap.cc:492] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:11.643348 10053 ts_tablet_manager.cc:1403] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.284s user 0.023s sys 0.004s
I20260430 07:58:11.637490 10009 raft_consensus.cc:2393] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 1: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
W20260430 07:58:11.574378 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 46782.5us
I20260430 07:58:11.535895 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:11.646541 10053 raft_consensus.cc:359] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.632697 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:11.647492 10053 raft_consensus.cc:385] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.648684 10053 raft_consensus.cc:740] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.674986 10053 consensus_queue.cc:260] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:11.681880 9809 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:58:11.545135 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:11.635313 9861 raft_consensus.cc:3060] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.640396 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:11.651984 10111 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:11.639834 10052 tablet_bootstrap.cc:654] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.574817 10375 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:11.667101 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:11.672747 9828 leader_election.cc:343] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.693450 10217 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.716344 10006 raft_consensus.cc:2393] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 1: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:11.729341 10053 ts_tablet_manager.cc:1434] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.086s user 0.010s sys 0.019s
I20260430 07:58:11.730756 9860 raft_consensus.cc:2393] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 1: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:11.740691 10053 tablet_bootstrap.cc:492] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:11.741899 10217 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.743142 10111 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.743759 10111 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.745800 10111 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:11.747360 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:11.748797 10007 raft_consensus.cc:2393] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 1: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:11.754910 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:11.756489 9978 leader_election.cc:343] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.769444 10415 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.769984 10415 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.768839 10054 tablet_bootstrap.cc:654] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.773977 10415 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.775548 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:11.776568 9934 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.777606 10052 tablet_bootstrap.cc:492] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:11.780364 10052 ts_tablet_manager.cc:1403] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.346s user 0.014s sys 0.008s
I20260430 07:58:11.783573 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:11.784335 10009 raft_consensus.cc:2468] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:11.785585 9828 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:11.789422 10379 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.801625 10379 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.803622 10053 tablet_bootstrap.cc:654] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.804674 10379 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.805111 9934 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:11.792160 10378 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260430 07:58:11.794262 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:11.807523 9937 raft_consensus.cc:2393] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 1: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:11.807632 9904 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:11.807925 10052 raft_consensus.cc:359] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.795023 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:11.808650 10052 raft_consensus.cc:385] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.808970 10052 raft_consensus.cc:740] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
W20260430 07:58:11.795280 9828 leader_election.cc:343] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.809433 10445 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:11.809549 9828 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:11.809635 10052 consensus_queue.cc:260] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.787267 10415 raft_consensus.cc:2804] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:11.810076 10445 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:11.810580 10415 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:11.810978 10415 raft_consensus.cc:3060] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:11.813067 9758 catalog_manager.cc:5671] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:11.815303 10453 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.815418 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:11.815860 10453 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.817996 10453 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.819628 10441 raft_consensus.cc:2749] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:11.786455 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:11.821051 9936 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:11.822000 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:11.822754 10052 ts_tablet_manager.cc:1434] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.042s user 0.006s sys 0.001s
I20260430 07:58:11.824651 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:11.825502 9904 leader_election.cc:343] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.826344 9977 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:11.828087 10052 tablet_bootstrap.cc:492] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:11.828469 10053 tablet_bootstrap.cc:492] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:11.828869 10230 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:11.829574 10230 raft_consensus.cc:697] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:11.830399 10230 consensus_queue.cc:237] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.828910 10053 ts_tablet_manager.cc:1403] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.095s user 0.013s sys 0.000s
I20260430 07:58:11.834537 10052 tablet_bootstrap.cc:654] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.834246 10053 raft_consensus.cc:359] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.834966 10053 raft_consensus.cc:385] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.835253 10053 raft_consensus.cc:740] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.836210 10053 consensus_queue.cc:260] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.839519 10053 ts_tablet_manager.cc:1434] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20260430 07:58:11.841593 9828 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:11.842707 10441 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260430 07:58:11.848632 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:11.849399 10007 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:11.851686 9977 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:11.852903 10440 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:11.853504 10440 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:11.853875 10440 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.854051 10453 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:11.854420 9904 leader_election.cc:343] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.854518 10453 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.857116 10453 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:11.859393 9903 leader_election.cc:343] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.852859 10053 tablet_bootstrap.cc:492] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:11.866874 9903 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:11.867547 10054 tablet_bootstrap.cc:492] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:11.868167 10054 ts_tablet_manager.cc:1403] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.226s user 0.008s sys 0.007s
I20260430 07:58:11.869092 10232 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:11.871423 10054 raft_consensus.cc:359] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.871881 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:11.872262 10054 raft_consensus.cc:385] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.872568 10054 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.873346 10054 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.872584 9861 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:11.874727 10052 tablet_bootstrap.cc:492] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:11.874527 10440 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.875262 10052 ts_tablet_manager.cc:1403] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.047s user 0.016s sys 0.024s
I20260430 07:58:11.878569 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:11.873719 9903 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:58:11.880430 9978 leader_election.cc:343] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.882110 10232 consensus_queue.cc:1048] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:11.884263 10054 ts_tablet_manager.cc:1434] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260430 07:58:11.886085 10054 tablet_bootstrap.cc:492] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:11.886031 9904 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:11.887223 10453 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:11.887594 10453 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:11.887909 10453 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.888428 10053 tablet_bootstrap.cc:654] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.888911 10052 raft_consensus.cc:359] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.889899 10052 raft_consensus.cc:385] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:11.890190 10052 raft_consensus.cc:740] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:11.890686 10379 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:11.894980 10052 consensus_queue.cc:260] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.897037 10052 ts_tablet_manager.cc:1434] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260430 07:58:11.897980 10052 tablet_bootstrap.cc:492] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:11.910339 10443 consensus_queue.cc:1048] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:11.894538 10379 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:11.911458 10379 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.913898 10052 tablet_bootstrap.cc:654] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:11.889602 10415 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.905912 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:11.924556 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:11.925318 9933 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.927606 10379 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.915400 10440 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:11.916386 10453 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:11.930699 10415 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.934124 9863 raft_consensus.cc:1240] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260430 07:58:11.928933 9759 catalog_manager.cc:5671] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
W20260430 07:58:11.939903 9903 leader_election.cc:343] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.945747 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:11.946430 9859 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:11.962853 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:11.965437 9904 leader_election.cc:343] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:11.952090 10415 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:11.970293 10379 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:11.973672 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:11.996527 10054 tablet_bootstrap.cc:654] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.002925 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:11.978268 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:11.978881 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:11.979907 10453 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.025506 9859 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
W20260430 07:58:12.022660 9903 leader_election.cc:343] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:12.024257 10008 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.027428 10053 tablet_bootstrap.cc:492] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:12.034590 10053 ts_tablet_manager.cc:1403] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.182s user 0.012s sys 0.011s
I20260430 07:58:11.981969 10415 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.047251 10415 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.052510 10008 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:12.068343 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:12.070770 9933 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:12.073086 10052 tablet_bootstrap.cc:492] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:12.074195 10052 ts_tablet_manager.cc:1403] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.176s user 0.015s sys 0.013s
I20260430 07:58:12.075467 10053 raft_consensus.cc:359] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.076319 10053 raft_consensus.cc:385] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.076567 10053 raft_consensus.cc:740] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
W20260430 07:58:12.077535 9828 leader_election.cc:343] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:12.077142 10053 consensus_queue.cc:260] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.077874 10052 raft_consensus.cc:359] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.079988 10052 raft_consensus.cc:385] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.080412 10052 raft_consensus.cc:740] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.081560 10052 consensus_queue.cc:260] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.083149 10053 ts_tablet_manager.cc:1434] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.048s user 0.006s sys 0.001s
I20260430 07:58:12.084367 10053 tablet_bootstrap.cc:492] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:12.084870 10052 ts_tablet_manager.cc:1434] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260430 07:58:12.085639 9977 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:12.086212 10052 tablet_bootstrap.cc:492] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:12.086741 10440 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:12.115471 10440 raft_consensus.cc:697] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:12.091766 9828 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:12.117615 10468 consensus_queue.cc:1059] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:12.118674 10378 raft_consensus.cc:3055] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:12.119086 10378 raft_consensus.cc:740] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:12.120090 10415 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:12.101461 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:12.121704 10378 consensus_queue.cc:260] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.119026 9904 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:12.123512 10453 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:12.124171 10378 raft_consensus.cc:3060] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:12.094744 9758 catalog_manager.cc:5671] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:12.128866 10453 raft_consensus.cc:697] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:12.131485 10453 consensus_queue.cc:237] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.118183 10440 consensus_queue.cc:237] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.146513 10052 tablet_bootstrap.cc:654] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.092767 10053 tablet_bootstrap.cc:654] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.152344 9903 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:12.153564 10446 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:12.154282 10446 raft_consensus.cc:697] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:58:12.154606 9828 consensus_peers.cc:597] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 -> Peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Couldn't send request to peer 48e81934d52a4e93be5171ceea193032. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:12.155086 10446 consensus_queue.cc:237] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.155854 10053 tablet_bootstrap.cc:492] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:12.156312 10053 ts_tablet_manager.cc:1403] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.072s user 0.011s sys 0.000s
I20260430 07:58:12.159431 10053 raft_consensus.cc:359] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.160158 10053 raft_consensus.cc:385] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.160429 10053 raft_consensus.cc:740] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.161096 10053 consensus_queue.cc:260] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.161988 10230 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:12.162551 10230 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.163511 10053 ts_tablet_manager.cc:1434] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260430 07:58:12.164780 10053 tablet_bootstrap.cc:492] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:12.166208 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:12.167582 10230 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:12.169248 9978 leader_election.cc:343] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
W20260430 07:58:12.180235 9828 leader_election.cc:343] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:12.178153 10277 consensus_queue.cc:1048] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260430 07:58:12.180706 9828 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:12.182180 10409 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:12.193398 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:12.196554 9977 leader_election.cc:343] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:12.196956 9977 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:12.202466 10082 raft_consensus.cc:2749] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:12.214143 10054 tablet_bootstrap.cc:492] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:12.214677 10054 ts_tablet_manager.cc:1403] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.329s user 0.048s sys 0.054s
I20260430 07:58:12.217311 10053 tablet_bootstrap.cc:654] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.230127 10054 raft_consensus.cc:359] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.231009 10054 raft_consensus.cc:385] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.231370 10054 raft_consensus.cc:740] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.232173 10054 consensus_queue.cc:260] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.222643 9758 catalog_manager.cc:5671] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:12.242110 10052 tablet_bootstrap.cc:492] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:12.242662 10052 ts_tablet_manager.cc:1403] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.157s user 0.014s sys 0.000s
I20260430 07:58:12.256506 10054 ts_tablet_manager.cc:1434] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.042s user 0.011s sys 0.004s
I20260430 07:58:12.255363 10468 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:12.266958 10468 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.277072 10054 tablet_bootstrap.cc:492] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:12.282889 9759 catalog_manager.cc:5671] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:12.285887 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:12.286281 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:12.286479 9860 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:12.286893 10008 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:12.287703 9904 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:12.288373 9759 catalog_manager.cc:5671] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:12.289139 10378 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:12.290045 10378 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:12.290474 10378 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.289374 10468 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.300563 10378 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.303234 10052 raft_consensus.cc:359] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.304328 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:12.305402 9860 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.308665 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:12.309454 10008 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.312155 9860 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:12.313483 9904 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:12.314913 10052 raft_consensus.cc:385] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.315219 10052 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.314640 10468 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:12.322187 10053 tablet_bootstrap.cc:492] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:12.322708 10053 ts_tablet_manager.cc:1403] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.158s user 0.015s sys 0.016s
I20260430 07:58:12.318998 10052 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.319123 10378 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.330497 10230 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:12.332803 10053 raft_consensus.cc:359] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.333478 10053 raft_consensus.cc:385] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.333706 10053 raft_consensus.cc:740] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.326903 10468 raft_consensus.cc:697] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:12.335024 10468 consensus_queue.cc:237] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.338321 10053 consensus_queue.cc:260] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.339233 10052 ts_tablet_manager.cc:1434] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.096s user 0.006s sys 0.001s
I20260430 07:58:12.339993 10008 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:12.341250 10053 ts_tablet_manager.cc:1434] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260430 07:58:12.341449 10054 tablet_bootstrap.cc:654] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.344702 10053 tablet_bootstrap.cc:492] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5: Bootstrap starting.
I20260430 07:58:12.343665 10052 tablet_bootstrap.cc:492] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:12.358073 10054 tablet_bootstrap.cc:492] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:12.358583 10054 ts_tablet_manager.cc:1403] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.082s user 0.011s sys 0.000s
I20260430 07:58:12.362243 10053 tablet_bootstrap.cc:654] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.373304 10477 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260430 07:58:12.374714 10054 raft_consensus.cc:359] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.375411 10054 raft_consensus.cc:385] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.375713 10054 raft_consensus.cc:740] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.377014 10054 consensus_queue.cc:260] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.380491 10052 tablet_bootstrap.cc:654] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.383739 10468 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:12.387425 10468 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.387953 10054 ts_tablet_manager.cc:1434] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20260430 07:58:12.389786 10468 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.390365 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:12.393121 9904 leader_election.cc:343] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:12.394098 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:12.395399 10008 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 0.
I20260430 07:58:12.396521 9903 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:12.397204 10232 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:12.397616 10232 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:12.397961 10232 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.408855 10232 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.415239 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:12.416038 10009 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.417346 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:12.419509 9904 leader_election.cc:343] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:12.409502 10054 tablet_bootstrap.cc:492] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:12.433862 10054 tablet_bootstrap.cc:654] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.434702 9758 catalog_manager.cc:5671] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:12.447999 10054 tablet_bootstrap.cc:492] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:12.448589 10054 ts_tablet_manager.cc:1403] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.003s
I20260430 07:58:12.450017 10009 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:12.451611 9903 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:12.449643 10232 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.452685 10378 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:12.462662 10415 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:12.463035 10054 raft_consensus.cc:359] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.463515 10378 raft_consensus.cc:697] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:12.463625 10415 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.464022 10054 raft_consensus.cc:385] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.465865 10415 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.466246 10054 raft_consensus.cc:740] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.467478 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:12.467442 10054 consensus_queue.cc:260] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.467903 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:12.468922 10009 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 1: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:12.464630 10378 consensus_queue.cc:237] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.470253 10054 ts_tablet_manager.cc:1434] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.021s user 0.002s sys 0.005s
I20260430 07:58:12.470714 9828 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:12.471670 10415 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.473210 10446 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:12.473757 10446 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.477548 10052 tablet_bootstrap.cc:492] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:12.478093 10052 ts_tablet_manager.cc:1403] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.135s user 0.017s sys 0.026s
I20260430 07:58:12.480975 10054 tablet_bootstrap.cc:492] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:12.481824 10053 tablet_bootstrap.cc:492] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5: No bootstrap required, opened a new log
I20260430 07:58:12.482285 10053 ts_tablet_manager.cc:1403] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5: Time spent bootstrapping tablet: real 0.138s user 0.012s sys 0.007s
I20260430 07:58:12.484153 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:12.485396 9904 leader_election.cc:343] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:12.485749 10446 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.485420 10053 raft_consensus.cc:359] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.486766 10053 raft_consensus.cc:385] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.487025 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:12.487036 10053 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Initialized, Role: FOLLOWER
W20260430 07:58:12.488180 9903 leader_election.cc:343] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260430 07:58:12.488013 10053 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.488603 9903 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:12.491439 10453 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:12.492898 10052 raft_consensus.cc:359] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.493849 10052 raft_consensus.cc:385] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.494380 10052 raft_consensus.cc:740] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.495838 10052 consensus_queue.cc:260] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.498623 10052 ts_tablet_manager.cc:1434] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260430 07:58:12.500761 10052 tablet_bootstrap.cc:492] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:12.502010 10415 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:12.502761 10053 ts_tablet_manager.cc:1434] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260430 07:58:12.508771 10052 tablet_bootstrap.cc:654] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.525453 10054 tablet_bootstrap.cc:654] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.526932 10440 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:12.527467 10440 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.530203 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:12.530891 9862 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:12.530712 10453 consensus_queue.cc:1048] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:12.530498 9758 catalog_manager.cc:5671] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:12.532968 9978 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:12.533910 10477 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:12.534312 10477 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:12.534667 10477 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
W20260430 07:58:12.538905 9978 consensus_peers.cc:597] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 -> Peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): Couldn't send request to peer 27f97a330f374849be58a168318c1ec6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:12.543776 10054 tablet_bootstrap.cc:492] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:12.544502 10054 ts_tablet_manager.cc:1403] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.006s
I20260430 07:58:12.550175 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:12.550912 9933 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:12.534672 10440 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:12.555603 10052 tablet_bootstrap.cc:492] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:12.556532 10052 ts_tablet_manager.cc:1403] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.056s user 0.019s sys 0.007s
I20260430 07:58:12.544646 10477 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.559309 10054 raft_consensus.cc:359] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.562160 10054 raft_consensus.cc:385] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.562461 10054 raft_consensus.cc:740] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.563220 10054 consensus_queue.cc:260] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.563583 10052 raft_consensus.cc:359] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.564224 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:12.564893 9860 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.565886 10052 raft_consensus.cc:385] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.566180 10052 raft_consensus.cc:740] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.566965 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:12.566883 10052 consensus_queue.cc:260] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.567543 9935 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.568131 10476 consensus_queue.cc:1048] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:12.569459 10052 ts_tablet_manager.cc:1434] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20260430 07:58:12.570835 10054 ts_tablet_manager.cc:1434] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260430 07:58:12.571950 10054 tablet_bootstrap.cc:492] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:12.572693 9860 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:12.573963 9978 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:12.575225 10052 tablet_bootstrap.cc:492] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:12.577154 10440 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:12.577818 10440 raft_consensus.cc:697] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:12.578974 10054 tablet_bootstrap.cc:654] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.578672 10440 consensus_queue.cc:237] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.592062 10054 tablet_bootstrap.cc:492] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:12.592545 10054 ts_tablet_manager.cc:1403] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.010s
I20260430 07:58:12.595197 10054 raft_consensus.cc:359] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.596022 10054 raft_consensus.cc:385] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.596292 10054 raft_consensus.cc:740] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.597069 10054 consensus_queue.cc:260] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.599157 10054 ts_tablet_manager.cc:1434] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260430 07:58:12.600281 10054 tablet_bootstrap.cc:492] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032: Bootstrap starting.
I20260430 07:58:12.600512 10232 consensus_queue.cc:1048] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260430 07:58:12.606400 9935 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:12.563995 10477 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:12.609338 10054 tablet_bootstrap.cc:654] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.628142 10005 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.616847 10052 tablet_bootstrap.cc:654] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.647713 10373 consensus_queue.cc:1048] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:12.669035 9761 catalog_manager.cc:5671] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:12.677484 10054 tablet_bootstrap.cc:492] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032: No bootstrap required, opened a new log
I20260430 07:58:12.678030 10054 ts_tablet_manager.cc:1403] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.023s
I20260430 07:58:12.681120 10054 raft_consensus.cc:359] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.681856 10054 raft_consensus.cc:385] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.682322 10054 raft_consensus.cc:740] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.682989 10054 consensus_queue.cc:260] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.685629 10054 ts_tablet_manager.cc:1434] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260430 07:58:12.688355 10453 consensus_queue.cc:1048] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:12.695134 10052 tablet_bootstrap.cc:492] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:12.695688 10052 ts_tablet_manager.cc:1403] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.121s user 0.023s sys 0.012s
I20260430 07:58:12.703963 10052 raft_consensus.cc:359] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.704757 10052 raft_consensus.cc:385] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.705001 10415 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:12.705094 10052 raft_consensus.cc:740] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.706043 10052 consensus_queue.cc:260] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.707381 10415 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.708822 10052 ts_tablet_manager.cc:1434] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260430 07:58:12.710803 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:12.711496 10008 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:12.711704 10052 tablet_bootstrap.cc:492] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6: Bootstrap starting.
I20260430 07:58:12.712714 9828 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:12.713919 10409 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:12.714283 10409 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:12.714624 10409 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.719620 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:12.720338 9933 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:12.720609 10052 tablet_bootstrap.cc:654] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6: Neither blocks nor log segments found. Creating new log.
I20260430 07:58:12.718788 10415 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.722783 10409 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.726234 10378 consensus_queue.cc:1048] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:12.726598 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:12.727174 10008 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.727881 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:12.728477 9933 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.735082 10008 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:12.736312 9828 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:12.737109 10415 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:12.739281 10009 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.748932 10415 raft_consensus.cc:697] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:12.750046 10415 consensus_queue.cc:237] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.756399 10409 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.778570 10409 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:12.779691 10409 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.783591 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:12.784267 9935 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:12.784299 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:12.785126 10008 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 0.
I20260430 07:58:12.785606 9828 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:12.793241 10277 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:12.793704 10277 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:12.794035 10277 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.799571 10052 tablet_bootstrap.cc:492] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6: No bootstrap required, opened a new log
I20260430 07:58:12.800156 10052 ts_tablet_manager.cc:1403] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6: Time spent bootstrapping tablet: real 0.089s user 0.017s sys 0.058s
I20260430 07:58:12.784709 9933 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:12.803185 10052 raft_consensus.cc:359] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.803977 10052 raft_consensus.cc:385] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260430 07:58:12.804329 10052 raft_consensus.cc:740] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Initialized, Role: FOLLOWER
I20260430 07:58:12.804831 10468 consensus_queue.cc:1048] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:12.804992 10052 consensus_queue.cc:260] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.806392 10277 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.807292 10052 ts_tablet_manager.cc:1434] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260430 07:58:12.792508 10409 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.812338 10277 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.814433 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:12.815080 10009 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.822572 10009 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:12.823586 9828 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:12.824507 10277 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:12.826195 10277 raft_consensus.cc:697] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:12.827191 10277 consensus_queue.cc:237] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.829986 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:12.829152 9759 catalog_manager.cc:5671] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:12.830617 9933 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.841892 9933 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:12.853915 9758 catalog_manager.cc:5671] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
W20260430 07:58:12.855693 9809 auto_rebalancer.cc:258] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260430 07:58:12.870396 10478 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:12.870870 10478 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.872687 10478 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:12.873705 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:12.874140 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:12.874542 9859 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:12.875262 9933 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 0.
I20260430 07:58:12.875672 9978 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:12.876639 10478 raft_consensus.cc:2804] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260430 07:58:12.877157 10478 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:12.877948 10478 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.885179 10478 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.887014 10478 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:12.887756 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:12.888221 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:12.888401 9859 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.888792 9933 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.892966 9859 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:12.893888 9933 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:12.894212 9978 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:12.895182 10478 raft_consensus.cc:2804] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader election won for term 1
I20260430 07:58:12.895637 10478 raft_consensus.cc:697] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:12.896410 10478 consensus_queue.cc:237] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.912917 9759 catalog_manager.cc:5671] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:12.975553 10277 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:12.976191 10277 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:12.978165 10277 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:12.979143 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:12.979233 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:12.981459 10005 raft_consensus.cc:2393] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 1: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:12.983071 9828 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:12.984166 10277 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:12.990403 10277 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:12.998253 10477 consensus_queue.cc:1048] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:13.010007 10478 consensus_queue.cc:1048] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:13.087005 10232 consensus_queue.cc:1048] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:13.092031 10277 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:13.093109 10277 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:13.099886 10277 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:13.100518 10232 consensus_queue.cc:1048] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:13.103214 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:13.103798 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:13.110844 9859 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 0 FOLLOWER]: Advancing to term 1
I20260430 07:58:13.110973 9828 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:13.120751 10277 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260430 07:58:13.133327 10277 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:13.145117 10217 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:13.276221 10217 consensus_queue.cc:1048] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:13.287969 10409 consensus_queue.cc:1048] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:13.334365 10478 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:13.347627 10480 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260430 07:58:13.551764 10480 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:13.552266 10480 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:13.554034 10480 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:13.554853 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:13.554963 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:13.555639 9859 raft_consensus.cc:2393] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:13.555611 9937 raft_consensus.cc:2468] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:13.556856 9977 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:13.557425 10480 raft_consensus.cc:2804] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:13.557723 10480 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:13.558023 10480 raft_consensus.cc:3060] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:13.564142 10480 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:13.567106 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:13.567853 9859 raft_consensus.cc:2393] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:13.569347 10480 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:13.570425 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:13.576107 9937 raft_consensus.cc:2468] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:13.577060 9977 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:13.578039 10480 raft_consensus.cc:2804] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:13.578522 10480 raft_consensus.cc:697] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:13.579331 10480 consensus_queue.cc:237] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:13.591482 9760 catalog_manager.cc:5671] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:13.794752 8670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260430 07:58:13.804157 10522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:58:13.807933 10525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260430 07:58:13.820547 10523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260430 07:58:13.821364 8670 server_base.cc:1061] running on GCE node
I20260430 07:58:13.826395 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260430 07:58:13.826817 8670 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260430 07:58:13.827119 8670 hybrid_clock.cc:648] HybridClock initialized: now 1777535893827099 us; error 0 us; skew 500 ppm
I20260430 07:58:13.828332 8670 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260430 07:58:13.833135 8670 webserver.cc:492] Webserver started at http://127.8.119.132:44237/ using document root <none> and password file <none>
I20260430 07:58:13.834342 8670 fs_manager.cc:362] Metadata directory not provided
I20260430 07:58:13.834702 8670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260430 07:58:13.835089 8670 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260430 07:58:13.837167 8670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-3-root/instance:
uuid: "d66db415f0f0402fa431349afbf59888"
format_stamp: "Formatted at 2026-04-30 07:58:13 on dist-test-slave-6ct4"
I20260430 07:58:13.844281 8670 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260430 07:58:13.850625 10530 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:13.851585 8670 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260430 07:58:13.851864 8670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-3-root
uuid: "d66db415f0f0402fa431349afbf59888"
format_stamp: "Formatted at 2026-04-30 07:58:13 on dist-test-slave-6ct4"
I20260430 07:58:13.852133 8670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260430 07:58:13.898877 8670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260430 07:58:13.900944 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260430 07:58:13.908622 8670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260430 07:58:13.912685 8670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260430 07:58:13.913108 8670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260430 07:58:13.913566 8670 ts_tablet_manager.cc:616] Registered 0 tablets
I20260430 07:58:13.913810 8670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260430 07:58:14.064136 9859 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:14.066376 10478 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260430 07:58:14.069092 10415 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:14.069732 10415 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:14.072299 10415 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:14.075444 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:14.076215 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:14.077514 9935 raft_consensus.cc:2410] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260430 07:58:14.079075 9828 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:14.090515 10217 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:14.224416 10595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.8.119.132:40723 every 8 connection(s)
I20260430 07:58:14.223434 8670 rpc_server.cc:307] RPC server started. Bound to: 127.8.119.132:40723
I20260430 07:58:14.263311 9933 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:14.292760 10478 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:14.318208 8670 auto_leader_rebalancer-test.cc:417] add a tserver: d66db415f0f0402fa431349afbf59888
I20260430 07:58:14.361032 10597 heartbeater.cc:344] Connected to a master server at 127.8.119.190:40689
I20260430 07:58:14.361613 10597 heartbeater.cc:461] Registering TS with master...
I20260430 07:58:14.362366 10597 heartbeater.cc:507] Master 127.8.119.190:40689 requested a full tablet report, sending...
I20260430 07:58:14.367689 9759 ts_manager.cc:194] Registered new tserver with Master: d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:14.371158 9759 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53852
I20260430 07:58:15.066195 10010 consensus_queue.cc:237] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:15.076722 9935 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:15.079088 9863 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:15.079978 10480 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:15.081413 10478 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:15.094907 10480 raft_consensus.cc:2955] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.096800 9935 raft_consensus.cc:2955] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.096990 9863 raft_consensus.cc:2955] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
W20260430 07:58:15.119443 9977 consensus_peers.cc:597] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2947a88351384118b009f71f871ab9c8. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:15.125506 9759 catalog_manager.cc:5671] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.151796 9937 consensus_queue.cc:237] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:15.168000 9863 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:15.169427 10598 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:15.172317 10010 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:15.173967 10468 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260430 07:58:15.269135 9903 consensus_peers.cc:597] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c10b37fe2e54d7a951545147ac2e381. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:15.276741 10468 raft_consensus.cc:2955] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.292734 9862 raft_consensus.cc:2955] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.309103 9759 catalog_manager.cc:5671] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:15.335935 10003 raft_consensus.cc:2955] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.357309 9863 consensus_queue.cc:237] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:15.374315 10597 heartbeater.cc:499] Master 127.8.119.190:40689 was elected leader, sending a full tablet report...
I20260430 07:58:15.374907 10010 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:15.381721 9935 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:15.382623 10623 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:15.383762 10415 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260430 07:58:15.394333 9828 consensus_peers.cc:597] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a66ec69e8f2c493fa90c40e25719d298. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:15.396577 10623 raft_consensus.cc:2955] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.402031 9935 raft_consensus.cc:2955] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.411566 10010 raft_consensus.cc:2955] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.424368 10003 consensus_queue.cc:237] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:15.435318 9759 catalog_manager.cc:5671] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 2 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
W20260430 07:58:15.447854 9977 consensus_peers.cc:597] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e8dad8acea649479d1e927cd32599fe. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:15.449663 9935 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260430 07:58:15.454841 9863 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260430 07:58:15.456867 10480 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:15.459476 10478 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:15.486604 10604 raft_consensus.cc:2955] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.488528 9862 raft_consensus.cc:2955] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.491182 9935 raft_consensus.cc:2955] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.506765 9760 catalog_manager.cc:5671] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 reported cstate change: config changed from index -1 to 3, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:15.562301 10632 ts_tablet_manager.cc:933] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:15.564945 10632 tablet_copy_client.cc:323] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.130:38147
I20260430 07:58:15.581017 9948 tablet_copy_service.cc:140] P 77e19ab653f540339c1c613fad1451b5: Received BeginTabletCopySession request for tablet 7c10b37fe2e54d7a951545147ac2e381 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:58726)
I20260430 07:58:15.581692 10635 ts_tablet_manager.cc:933] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:15.581924 9948 tablet_copy_service.cc:161] P 77e19ab653f540339c1c613fad1451b5: Beginning new tablet copy session on tablet 7c10b37fe2e54d7a951545147ac2e381 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:58726: session id = d66db415f0f0402fa431349afbf59888-7c10b37fe2e54d7a951545147ac2e381
I20260430 07:58:15.585840 10635 tablet_copy_client.cc:323] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.131:39205
I20260430 07:58:15.594763 9948 tablet_copy_source_session.cc:215] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:15.599807 10632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c10b37fe2e54d7a951545147ac2e381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:15.612128 10022 tablet_copy_service.cc:140] P 48e81934d52a4e93be5171ceea193032: Received BeginTabletCopySession request for tablet 2947a88351384118b009f71f871ab9c8 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:53754)
I20260430 07:58:15.612735 10022 tablet_copy_service.cc:161] P 48e81934d52a4e93be5171ceea193032: Beginning new tablet copy session on tablet 2947a88351384118b009f71f871ab9c8 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:53754: session id = d66db415f0f0402fa431349afbf59888-2947a88351384118b009f71f871ab9c8
I20260430 07:58:15.618428 10022 tablet_copy_source_session.cc:215] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:15.620553 10632 tablet_copy_client.cc:806] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:15.621312 10632 tablet_copy_client.cc:670] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:15.622027 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2947a88351384118b009f71f871ab9c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:15.627375 10632 tablet_copy_client.cc:538] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:15.638118 10632 tablet_bootstrap.cc:492] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:15.642716 10635 tablet_copy_client.cc:806] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:15.643373 10635 tablet_copy_client.cc:670] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:15.648840 10635 tablet_copy_client.cc:538] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:15.662631 10635 tablet_bootstrap.cc:492] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:15.698838 10632 tablet_bootstrap.cc:492] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:15.700217 10632 tablet_bootstrap.cc:492] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:15.701121 10632 ts_tablet_manager.cc:1403] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.063s user 0.021s sys 0.013s
I20260430 07:58:15.714416 10632 raft_consensus.cc:359] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:15.715375 10632 raft_consensus.cc:740] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:15.716097 10632 consensus_queue.cc:260] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:15.718822 10635 tablet_bootstrap.cc:492] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:15.719995 10635 tablet_bootstrap.cc:492] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:15.720927 10635 ts_tablet_manager.cc:1403] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.059s user 0.035s sys 0.016s
I20260430 07:58:15.725816 10635 raft_consensus.cc:359] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:15.726599 10632 ts_tablet_manager.cc:1434] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.025s user 0.010s sys 0.003s
I20260430 07:58:15.726719 10635 raft_consensus.cc:740] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:15.727447 10635 consensus_queue.cc:260] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:15.729941 9948 tablet_copy_service.cc:342] P 77e19ab653f540339c1c613fad1451b5: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-7c10b37fe2e54d7a951545147ac2e381 received from {username='slave'} at 127.0.0.1:58726
I20260430 07:58:15.730986 9948 tablet_copy_service.cc:434] P 77e19ab653f540339c1c613fad1451b5: ending tablet copy session d66db415f0f0402fa431349afbf59888-7c10b37fe2e54d7a951545147ac2e381 on tablet 7c10b37fe2e54d7a951545147ac2e381 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:15.745913 10635 ts_tablet_manager.cc:1434] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.025s user 0.012s sys 0.008s
I20260430 07:58:15.747556 10022 tablet_copy_service.cc:342] P 48e81934d52a4e93be5171ceea193032: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-2947a88351384118b009f71f871ab9c8 received from {username='slave'} at 127.0.0.1:53754
I20260430 07:58:15.748008 10022 tablet_copy_service.cc:434] P 48e81934d52a4e93be5171ceea193032: ending tablet copy session d66db415f0f0402fa431349afbf59888-2947a88351384118b009f71f871ab9c8 on tablet 2947a88351384118b009f71f871ab9c8 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:15.798115 10635 ts_tablet_manager.cc:933] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679)
I20260430 07:58:15.800004 10635 tablet_copy_client.cc:323] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.129:36679
I20260430 07:58:15.814229 9873 tablet_copy_service.cc:140] P 27f97a330f374849be58a168318c1ec6: Received BeginTabletCopySession request for tablet a66ec69e8f2c493fa90c40e25719d298 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:41094)
I20260430 07:58:15.814838 9873 tablet_copy_service.cc:161] P 27f97a330f374849be58a168318c1ec6: Beginning new tablet copy session on tablet a66ec69e8f2c493fa90c40e25719d298 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:41094: session id = d66db415f0f0402fa431349afbf59888-a66ec69e8f2c493fa90c40e25719d298
I20260430 07:58:15.823140 9873 tablet_copy_source_session.cc:215] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:15.827510 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66ec69e8f2c493fa90c40e25719d298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:15.845769 10635 tablet_copy_client.cc:806] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:15.846449 10635 tablet_copy_client.cc:670] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:15.850557 10635 tablet_copy_client.cc:538] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:15.868687 10635 tablet_bootstrap.cc:492] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:15.894016 10635 tablet_bootstrap.cc:492] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:15.896674 10635 tablet_bootstrap.cc:492] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:15.897644 10635 ts_tablet_manager.cc:1403] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.029s user 0.026s sys 0.005s
I20260430 07:58:15.901188 10635 raft_consensus.cc:359] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:15.904083 10635 raft_consensus.cc:740] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:15.905237 10635 consensus_queue.cc:260] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:15.908115 10635 ts_tablet_manager.cc:1434] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20260430 07:58:15.909829 9873 tablet_copy_service.cc:342] P 27f97a330f374849be58a168318c1ec6: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-a66ec69e8f2c493fa90c40e25719d298 received from {username='slave'} at 127.0.0.1:41094
I20260430 07:58:15.910507 9873 tablet_copy_service.cc:434] P 27f97a330f374849be58a168318c1ec6: ending tablet copy session d66db415f0f0402fa431349afbf59888-a66ec69e8f2c493fa90c40e25719d298 on tablet a66ec69e8f2c493fa90c40e25719d298 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:16.040855 10635 ts_tablet_manager.cc:933] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:16.042793 10635 tablet_copy_client.cc:323] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.131:39205
I20260430 07:58:16.045419 10022 tablet_copy_service.cc:140] P 48e81934d52a4e93be5171ceea193032: Received BeginTabletCopySession request for tablet 2e8dad8acea649479d1e927cd32599fe from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:53754)
I20260430 07:58:16.045853 10022 tablet_copy_service.cc:161] P 48e81934d52a4e93be5171ceea193032: Beginning new tablet copy session on tablet 2e8dad8acea649479d1e927cd32599fe from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:53754: session id = d66db415f0f0402fa431349afbf59888-2e8dad8acea649479d1e927cd32599fe
I20260430 07:58:16.053306 10022 tablet_copy_source_session.cc:215] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:16.056895 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e8dad8acea649479d1e927cd32599fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:16.074208 10635 tablet_copy_client.cc:806] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:16.074903 10635 tablet_copy_client.cc:670] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:16.079828 10635 tablet_copy_client.cc:538] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:16.088384 10635 tablet_bootstrap.cc:492] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:16.110453 10568 raft_consensus.cc:1217] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:16.111294 10635 tablet_bootstrap.cc:492] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:16.112332 10635 tablet_bootstrap.cc:492] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:16.113014 10635 ts_tablet_manager.cc:1403] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.025s user 0.025s sys 0.000s
I20260430 07:58:16.124632 10635 raft_consensus.cc:359] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:16.125468 10635 raft_consensus.cc:740] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:16.126565 10635 consensus_queue.cc:260] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:16.128527 10635 ts_tablet_manager.cc:1434] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260430 07:58:16.150339 10022 tablet_copy_service.cc:342] P 48e81934d52a4e93be5171ceea193032: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-2e8dad8acea649479d1e927cd32599fe received from {username='slave'} at 127.0.0.1:53754
I20260430 07:58:16.150841 10022 tablet_copy_service.cc:434] P 48e81934d52a4e93be5171ceea193032: ending tablet copy session d66db415f0f0402fa431349afbf59888-2e8dad8acea649479d1e927cd32599fe on tablet 2e8dad8acea649479d1e927cd32599fe with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:16.165381 10568 raft_consensus.cc:1217] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:16.235481 10568 raft_consensus.cc:1217] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260430 07:58:16.552067 10568 raft_consensus.cc:1217] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260430 07:58:16.663833 10596 raft_consensus.cc:1064] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:16.666663 10647 raft_consensus.cc:1064] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:16.668063 10596 consensus_queue.cc:237] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:16.670884 10647 consensus_queue.cc:237] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:16.683797 10567 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:16.684247 10010 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:16.685827 10598 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:16.687392 10644 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:16.697999 9863 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:16.699879 10598 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:16.707536 9935 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:16.708030 10567 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:16.710052 9862 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:16.716431 10480 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:16.717825 10605 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:16.728540 10650 raft_consensus.cc:2955] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.730711 10008 raft_consensus.cc:2955] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.738612 9863 raft_consensus.cc:2955] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.745213 10567 raft_consensus.cc:2955] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.760795 9758 catalog_manager.cc:5671] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.768189 10623 raft_consensus.cc:1064] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:16.770826 10623 consensus_queue.cc:237] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:16.782606 10654 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:16.786727 10010 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260430 07:58:16.786981 10647 raft_consensus.cc:2955] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.789690 10415 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:16.796679 9937 raft_consensus.cc:2955] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.812269 10568 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 LEARNER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260430 07:58:16.817914 9935 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260430 07:58:16.817589 10567 raft_consensus.cc:2955] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.837453 10623 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:16.842375 9859 raft_consensus.cc:2955] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.853884 10623 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:16.941092 10661 raft_consensus.cc:2955] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.953373 10005 raft_consensus.cc:2955] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.987092 10567 raft_consensus.cc:2955] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:16.989655 9761 catalog_manager.cc:5671] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:17.003309 9935 consensus_queue.cc:237] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:17.009127 9937 raft_consensus.cc:2955] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.013437 10568 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:17.014824 10649 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:17.015821 9858 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:17.017525 10644 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:17.025802 10596 raft_consensus.cc:2955] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.027009 10568 raft_consensus.cc:2955] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.035902 9862 raft_consensus.cc:2955] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.036962 9759 catalog_manager.cc:5671] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:17.058104 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 7c10b37fe2e54d7a951545147ac2e381 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:17.066675 9862 consensus_queue.cc:237] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:17.070678 9760 catalog_manager.cc:5671] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.080756 10568 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260430 07:58:17.085515 10005 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260430 07:58:17.087088 10669 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:17.088575 10415 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:17.095543 10002 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c10b37fe2e54d7a951545147ac2e381 with delete_type TABLET_DATA_TOMBSTONED (TS 48e81934d52a4e93be5171ceea193032 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44006
I20260430 07:58:17.105192 10605 raft_consensus.cc:1064] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:17.106573 10661 raft_consensus.cc:2955] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.107846 10605 consensus_queue.cc:237] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:17.108920 10670 tablet_replica.cc:333] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032: stopping tablet replica
I20260430 07:58:17.109953 10670 raft_consensus.cc:2243] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:17.111528 10670 raft_consensus.cc:2272] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:17.120869 10567 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 LEARNER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260430 07:58:17.120966 10568 raft_consensus.cc:2955] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.122696 9858 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260430 07:58:17.125244 10605 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:17.126564 10480 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:17.129022 10005 raft_consensus.cc:2955] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.132510 9748 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet a66ec69e8f2c493fa90c40e25719d298 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:17.138578 9937 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260430 07:58:17.141229 10670 ts_tablet_manager.cc:1916] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:17.143054 10480 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:17.148027 9758 catalog_manager.cc:5671] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New cstate: current_term: 2 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.163872 10605 raft_consensus.cc:2955] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.168191 9938 raft_consensus.cc:2955] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.181639 10670 ts_tablet_manager.cc:1929] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:17.182264 10670 log.cc:1199] T 7c10b37fe2e54d7a951545147ac2e381 P 48e81934d52a4e93be5171ceea193032: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-2-root/wals/7c10b37fe2e54d7a951545147ac2e381
I20260430 07:58:17.181838 9863 raft_consensus.cc:2955] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.183919 9747 catalog_manager.cc:5002] TS 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): tablet 7c10b37fe2e54d7a951545147ac2e381 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:17.185947 10568 raft_consensus.cc:2955] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.187239 9924 tablet_service.cc:1558] Processing DeleteTablet for tablet a66ec69e8f2c493fa90c40e25719d298 with delete_type TABLET_DATA_TOMBSTONED (TS 77e19ab653f540339c1c613fad1451b5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35252
I20260430 07:58:17.206029 9758 catalog_manager.cc:5671] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 3 to 4, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.238662 10675 tablet_replica.cc:333] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5: stopping tablet replica
I20260430 07:58:17.238839 10003 consensus_queue.cc:237] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:17.239714 10675 raft_consensus.cc:2243] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:17.241102 10675 raft_consensus.cc:2272] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:17.244294 10568 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:17.244438 10675 ts_tablet_manager.cc:1916] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:17.245386 9863 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:17.245663 10605 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:58:17.247407 10647 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:58:17.253523 10480 raft_consensus.cc:2955] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.256148 10568 raft_consensus.cc:2955] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.262028 10675 ts_tablet_manager.cc:1929] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260430 07:58:17.262643 10675 log.cc:1199] T a66ec69e8f2c493fa90c40e25719d298 P 77e19ab653f540339c1c613fad1451b5: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-1-root/wals/a66ec69e8f2c493fa90c40e25719d298
I20260430 07:58:17.264632 9747 catalog_manager.cc:5002] TS 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): tablet a66ec69e8f2c493fa90c40e25719d298 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:17.267187 9863 raft_consensus.cc:2955] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.270435 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 2e8dad8acea649479d1e927cd32599fe with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:17.281551 9761 catalog_manager.cc:5671] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 4 to 5, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.297432 9924 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e8dad8acea649479d1e927cd32599fe with delete_type TABLET_DATA_TOMBSTONED (TS 77e19ab653f540339c1c613fad1451b5 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35252
I20260430 07:58:17.299321 10675 tablet_replica.cc:333] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5: stopping tablet replica
I20260430 07:58:17.315047 10675 raft_consensus.cc:2243] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:17.316846 10675 raft_consensus.cc:2272] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:17.321542 10675 ts_tablet_manager.cc:1916] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:17.336167 10675 ts_tablet_manager.cc:1929] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260430 07:58:17.336849 10675 log.cc:1199] T 2e8dad8acea649479d1e927cd32599fe P 77e19ab653f540339c1c613fad1451b5: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-1-root/wals/2e8dad8acea649479d1e927cd32599fe
I20260430 07:58:17.339649 9747 catalog_manager.cc:5002] TS 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): tablet 2e8dad8acea649479d1e927cd32599fe (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:17.360484 10005 consensus_queue.cc:237] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:17.365818 10568 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:17.367069 10654 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:17.367990 9938 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:17.369515 10654 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:17.376279 10655 raft_consensus.cc:2955] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.377871 10568 raft_consensus.cc:2955] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.381877 9938 raft_consensus.cc:2955] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.391803 9760 catalog_manager.cc:5671] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:17.405606 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 2947a88351384118b009f71f871ab9c8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:17.415427 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet 2947a88351384118b009f71f871ab9c8 with delete_type TABLET_DATA_TOMBSTONED (TS 27f97a330f374849be58a168318c1ec6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37818
I20260430 07:58:17.421097 10676 tablet_replica.cc:333] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6: stopping tablet replica
I20260430 07:58:17.422209 10676 raft_consensus.cc:2243] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:17.422979 10676 raft_consensus.cc:2272] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:17.426420 10676 ts_tablet_manager.cc:1916] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:17.440482 10676 ts_tablet_manager.cc:1929] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:17.440845 10676 log.cc:1199] T 2947a88351384118b009f71f871ab9c8 P 27f97a330f374849be58a168318c1ec6: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root/wals/2947a88351384118b009f71f871ab9c8
I20260430 07:58:17.442257 9748 catalog_manager.cc:5002] TS 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): tablet 2947a88351384118b009f71f871ab9c8 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:18.490108 9938 consensus_queue.cc:237] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:18.496949 10006 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:18.496949 9862 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:18.499199 10680 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260430 07:58:18.498186 9903 consensus_peers.cc:597] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02a63423ae464eb7b415a38419524150. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:18.501550 10681 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:18.511374 10681 raft_consensus.cc:2955] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.514096 9858 raft_consensus.cc:2955] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.514829 10004 raft_consensus.cc:2955] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.533608 9938 consensus_queue.cc:237] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:18.536087 9761 catalog_manager.cc:5671] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.545396 10004 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260430 07:58:18.545542 9903 consensus_peers.cc:597] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b978c5a9ab41402c806352f5ace14295. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:18.547962 10681 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:18.549054 9858 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:18.552551 10681 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:18.560065 10680 raft_consensus.cc:2955] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.564428 10004 raft_consensus.cc:2955] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.591634 9938 consensus_queue.cc:237] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:18.591005 9858 raft_consensus.cc:2955] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.613250 9858 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260430 07:58:18.614115 9903 consensus_peers.cc:597] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3028e67ff7d948ee8accee91fbcb70c3. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:18.614773 10681 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:18.617208 9758 catalog_manager.cc:5671] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.620231 10004 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:18.638185 10681 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:18.658670 10679 raft_consensus.cc:2955] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.660605 10004 raft_consensus.cc:2955] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.669891 9858 raft_consensus.cc:2955] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.673682 10008 consensus_queue.cc:237] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:18.677956 9760 catalog_manager.cc:5671] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:18.702196 9858 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260430 07:58:18.702524 9977 consensus_peers.cc:597] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e18e051a7225424ebb64aedb21bb97df. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:18.702196 9938 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:18.703842 10655 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:18.704803 10654 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:18.714521 10654 raft_consensus.cc:2955] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.716097 9937 raft_consensus.cc:2955] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.719468 9858 raft_consensus.cc:2955] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.735852 9758 catalog_manager.cc:5671] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:18.899860 10701 ts_tablet_manager.cc:933] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:18.904734 10701 tablet_copy_client.cc:323] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.130:38147
I20260430 07:58:18.907202 9948 tablet_copy_service.cc:140] P 77e19ab653f540339c1c613fad1451b5: Received BeginTabletCopySession request for tablet 02a63423ae464eb7b415a38419524150 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:58726)
I20260430 07:58:18.907763 9948 tablet_copy_service.cc:161] P 77e19ab653f540339c1c613fad1451b5: Beginning new tablet copy session on tablet 02a63423ae464eb7b415a38419524150 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:58726: session id = d66db415f0f0402fa431349afbf59888-02a63423ae464eb7b415a38419524150
I20260430 07:58:18.916270 9948 tablet_copy_source_session.cc:215] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:18.919745 10701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02a63423ae464eb7b415a38419524150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:18.941009 10701 tablet_copy_client.cc:806] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:18.941704 10701 tablet_copy_client.cc:670] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:18.949745 10701 tablet_copy_client.cc:538] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:18.962688 10701 tablet_bootstrap.cc:492] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:18.987568 10701 tablet_bootstrap.cc:492] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:18.988483 10701 tablet_bootstrap.cc:492] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:18.989171 10701 ts_tablet_manager.cc:1403] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.005s
I20260430 07:58:18.992323 10701 raft_consensus.cc:359] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:18.993147 10701 raft_consensus.cc:740] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:18.994227 10701 consensus_queue.cc:260] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:18.997421 10701 ts_tablet_manager.cc:1434] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20260430 07:58:19.000420 9948 tablet_copy_service.cc:342] P 77e19ab653f540339c1c613fad1451b5: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-02a63423ae464eb7b415a38419524150 received from {username='slave'} at 127.0.0.1:58726
I20260430 07:58:19.001259 9948 tablet_copy_service.cc:434] P 77e19ab653f540339c1c613fad1451b5: ending tablet copy session d66db415f0f0402fa431349afbf59888-02a63423ae464eb7b415a38419524150 on tablet 02a63423ae464eb7b415a38419524150 with peer d66db415f0f0402fa431349afbf59888
W20260430 07:58:19.017396 9792 debug-util.cc:398] Leaking SignalData structure 0x7b0800035c40 after lost signal to thread 8673
W20260430 07:58:19.018821 9792 debug-util.cc:398] Leaking SignalData structure 0x7b08000649e0 after lost signal to thread 9795
W20260430 07:58:19.020084 9792 debug-util.cc:398] Leaking SignalData structure 0x7b08002a6e40 after lost signal to thread 9887
W20260430 07:58:19.021257 9792 debug-util.cc:398] Leaking SignalData structure 0x7b08002b7760 after lost signal to thread 9962
W20260430 07:58:19.022619 9792 debug-util.cc:398] Leaking SignalData structure 0x7b0800312740 after lost signal to thread 10036
W20260430 07:58:19.023666 9792 debug-util.cc:398] Leaking SignalData structure 0x7b08002f13e0 after lost signal to thread 10595
I20260430 07:58:19.127074 10701 ts_tablet_manager.cc:933] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:19.141676 10706 ts_tablet_manager.cc:933] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:19.144392 10701 tablet_copy_client.cc:323] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.131:39205
I20260430 07:58:19.147049 10022 tablet_copy_service.cc:140] P 48e81934d52a4e93be5171ceea193032: Received BeginTabletCopySession request for tablet e18e051a7225424ebb64aedb21bb97df from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:53754)
I20260430 07:58:19.147480 10706 tablet_copy_client.cc:323] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.130:38147
I20260430 07:58:19.147655 10022 tablet_copy_service.cc:161] P 48e81934d52a4e93be5171ceea193032: Beginning new tablet copy session on tablet e18e051a7225424ebb64aedb21bb97df from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:53754: session id = d66db415f0f0402fa431349afbf59888-e18e051a7225424ebb64aedb21bb97df
I20260430 07:58:19.149237 9948 tablet_copy_service.cc:140] P 77e19ab653f540339c1c613fad1451b5: Received BeginTabletCopySession request for tablet b978c5a9ab41402c806352f5ace14295 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:58726)
I20260430 07:58:19.149833 9948 tablet_copy_service.cc:161] P 77e19ab653f540339c1c613fad1451b5: Beginning new tablet copy session on tablet b978c5a9ab41402c806352f5ace14295 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:58726: session id = d66db415f0f0402fa431349afbf59888-b978c5a9ab41402c806352f5ace14295
I20260430 07:58:19.158908 9948 tablet_copy_source_session.cc:215] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:19.164611 10706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b978c5a9ab41402c806352f5ace14295. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:19.171710 10022 tablet_copy_source_session.cc:215] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:19.175920 10701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e18e051a7225424ebb64aedb21bb97df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:19.215587 10701 tablet_copy_client.cc:806] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:19.216251 10701 tablet_copy_client.cc:670] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:19.217168 10706 tablet_copy_client.cc:806] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:19.217882 10706 tablet_copy_client.cc:670] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:19.222630 10706 tablet_copy_client.cc:538] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:19.227293 10701 tablet_copy_client.cc:538] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:19.239542 10701 tablet_bootstrap.cc:492] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:19.247457 10706 tablet_bootstrap.cc:492] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:19.255420 10708 ts_tablet_manager.cc:933] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:19.371217 10708 tablet_copy_client.cc:323] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.130:38147
I20260430 07:58:19.374140 9948 tablet_copy_service.cc:140] P 77e19ab653f540339c1c613fad1451b5: Received BeginTabletCopySession request for tablet 3028e67ff7d948ee8accee91fbcb70c3 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:58726)
I20260430 07:58:19.374765 9948 tablet_copy_service.cc:161] P 77e19ab653f540339c1c613fad1451b5: Beginning new tablet copy session on tablet 3028e67ff7d948ee8accee91fbcb70c3 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:58726: session id = d66db415f0f0402fa431349afbf59888-3028e67ff7d948ee8accee91fbcb70c3
I20260430 07:58:19.387174 9948 tablet_copy_source_session.cc:215] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:19.400666 10708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3028e67ff7d948ee8accee91fbcb70c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:19.402155 10701 tablet_bootstrap.cc:492] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:19.403388 10701 tablet_bootstrap.cc:492] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:19.404603 10701 ts_tablet_manager.cc:1403] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.165s user 0.025s sys 0.023s
I20260430 07:58:19.393630 10706 tablet_bootstrap.cc:492] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:19.410359 10706 tablet_bootstrap.cc:492] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:19.411345 10706 ts_tablet_manager.cc:1403] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.164s user 0.036s sys 0.015s
I20260430 07:58:19.416318 10706 raft_consensus.cc:359] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:19.408651 10701 raft_consensus.cc:359] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:19.418862 10706 raft_consensus.cc:740] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:19.419938 10706 consensus_queue.cc:260] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:19.419178 10701 raft_consensus.cc:740] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:19.421777 10701 consensus_queue.cc:260] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:19.424247 10701 ts_tablet_manager.cc:1434] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.019s user 0.002s sys 0.003s
I20260430 07:58:19.425915 10022 tablet_copy_service.cc:342] P 48e81934d52a4e93be5171ceea193032: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-e18e051a7225424ebb64aedb21bb97df received from {username='slave'} at 127.0.0.1:53754
I20260430 07:58:19.426367 10022 tablet_copy_service.cc:434] P 48e81934d52a4e93be5171ceea193032: ending tablet copy session d66db415f0f0402fa431349afbf59888-e18e051a7225424ebb64aedb21bb97df on tablet e18e051a7225424ebb64aedb21bb97df with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:19.427404 10706 ts_tablet_manager.cc:1434] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.016s user 0.003s sys 0.005s
I20260430 07:58:19.428735 10708 tablet_copy_client.cc:806] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:19.429656 9948 tablet_copy_service.cc:342] P 77e19ab653f540339c1c613fad1451b5: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-b978c5a9ab41402c806352f5ace14295 received from {username='slave'} at 127.0.0.1:58726
I20260430 07:58:19.433554 9948 tablet_copy_service.cc:434] P 77e19ab653f540339c1c613fad1451b5: ending tablet copy session d66db415f0f0402fa431349afbf59888-b978c5a9ab41402c806352f5ace14295 on tablet b978c5a9ab41402c806352f5ace14295 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:19.433549 10708 tablet_copy_client.cc:670] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:19.439963 10708 tablet_copy_client.cc:538] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:19.452092 10708 tablet_bootstrap.cc:492] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:19.480337 10708 tablet_bootstrap.cc:492] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:19.481392 10708 tablet_bootstrap.cc:492] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:19.482149 10708 ts_tablet_manager.cc:1403] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.006s
I20260430 07:58:19.488621 10708 raft_consensus.cc:359] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:19.489640 10708 raft_consensus.cc:740] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:19.490391 10708 consensus_queue.cc:260] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:19.492925 10708 ts_tablet_manager.cc:1434] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260430 07:58:19.494727 9948 tablet_copy_service.cc:342] P 77e19ab653f540339c1c613fad1451b5: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-3028e67ff7d948ee8accee91fbcb70c3 received from {username='slave'} at 127.0.0.1:58726
I20260430 07:58:19.495186 9948 tablet_copy_service.cc:434] P 77e19ab653f540339c1c613fad1451b5: ending tablet copy session d66db415f0f0402fa431349afbf59888-3028e67ff7d948ee8accee91fbcb70c3 on tablet 3028e67ff7d948ee8accee91fbcb70c3 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:19.519371 10568 raft_consensus.cc:1217] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260430 07:58:19.613335 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 48648.4us
I20260430 07:58:19.708015 10568 raft_consensus.cc:1217] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:19.786360 10567 raft_consensus.cc:1217] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:19.788952 10568 raft_consensus.cc:1217] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:20.014343 10680 raft_consensus.cc:1064] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:20.017881 10680 consensus_queue.cc:237] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:20.027283 9862 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.027832 10568 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.030527 10644 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:20.030964 10004 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.039280 10644 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:20.043574 10644 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:20.052541 10680 raft_consensus.cc:2955] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.068632 9860 raft_consensus.cc:2955] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.073433 10568 raft_consensus.cc:2955] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.068921 10009 raft_consensus.cc:2955] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.099144 10721 raft_consensus.cc:1064] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:20.105459 9760 catalog_manager.cc:5671] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:20.104660 10721 consensus_queue.cc:237] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:20.154268 10568 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.157828 9862 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.160434 9935 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.156723 10722 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260430 07:58:20.163791 10711 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:20.165172 10721 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:20.197683 10720 raft_consensus.cc:2955] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.202085 9858 raft_consensus.cc:2955] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.211102 10644 raft_consensus.cc:1064] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:20.214309 10644 consensus_queue.cc:237] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:20.257599 9935 consensus_queue.cc:237] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:20.270848 9934 raft_consensus.cc:2955] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.275306 10568 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:20.280836 9761 catalog_manager.cc:5671] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.292210 10568 raft_consensus.cc:2955] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.334142 10004 consensus_queue.cc:237] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:20.338984 10009 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:20.367076 9858 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:20.369017 10721 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:20.375113 10567 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:20.386229 9860 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.386294 10711 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:20.389503 10683 raft_consensus.cc:2955] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.399051 10679 raft_consensus.cc:1064] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:20.410014 10679 consensus_queue.cc:237] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:20.399684 10716 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:20.398921 10567 raft_consensus.cc:2955] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.401528 9860 raft_consensus.cc:2955] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
W20260430 07:58:20.403043 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 58850.8us
I20260430 07:58:20.421617 10680 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:20.424489 10568 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.434135 10009 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.435621 10681 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:20.447038 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet e18e051a7225424ebb64aedb21bb97df with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:20.466701 10716 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:20.477128 9858 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.483071 10679 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:20.493585 10679 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:20.497860 10566 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.497097 10719 raft_consensus.cc:2955] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.501322 9759 catalog_manager.cc:5671] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.512655 10003 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:20.527277 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 02a63423ae464eb7b415a38419524150 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:20.535332 10680 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:20.536715 9760 catalog_manager.cc:5671] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:20.537495 10568 raft_consensus.cc:2955] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.540454 10719 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:20.548964 10004 raft_consensus.cc:2955] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.571348 10680 raft_consensus.cc:2955] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.579492 10004 raft_consensus.cc:2955] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.588109 10718 raft_consensus.cc:2955] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.602785 10004 raft_consensus.cc:2955] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.612449 10566 raft_consensus.cc:2955] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.616149 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet 02a63423ae464eb7b415a38419524150 with delete_type TABLET_DATA_TOMBSTONED (TS 27f97a330f374849be58a168318c1ec6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37818
I20260430 07:58:20.614903 10567 raft_consensus.cc:2955] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.624639 9758 catalog_manager.cc:5671] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:20.628754 9860 raft_consensus.cc:2955] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.631826 9924 tablet_service.cc:1558] Processing DeleteTablet for tablet e18e051a7225424ebb64aedb21bb97df with delete_type TABLET_DATA_TOMBSTONED (TS 77e19ab653f540339c1c613fad1451b5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35252
I20260430 07:58:20.634474 9863 raft_consensus.cc:2955] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.707753 10744 tablet_replica.cc:333] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5: stopping tablet replica
I20260430 07:58:20.709589 10744 raft_consensus.cc:2243] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:20.711130 10743 tablet_replica.cc:333] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6: stopping tablet replica
I20260430 07:58:20.724396 9761 catalog_manager.cc:5671] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.744802 10744 raft_consensus.cc:2272] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:20.756376 10743 raft_consensus.cc:2243] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:20.760778 10743 raft_consensus.cc:2272] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:20.773437 10744 ts_tablet_manager.cc:1916] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:20.783021 9938 consensus_queue.cc:237] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:20.783129 9935 consensus_queue.cc:237] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:20.793509 10743 ts_tablet_manager.cc:1916] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:20.799206 9858 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:20.811058 10008 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:20.814160 10567 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:20.814249 10566 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260430 07:58:20.816442 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 72454.3us
I20260430 07:58:20.835289 10718 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:20.837177 10680 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:20.842471 10744 ts_tablet_manager.cc:1929] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:20.843227 10744 log.cc:1199] T e18e051a7225424ebb64aedb21bb97df P 77e19ab653f540339c1c613fad1451b5: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-1-root/wals/e18e051a7225424ebb64aedb21bb97df
I20260430 07:58:20.851699 9747 catalog_manager.cc:5002] TS 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): tablet e18e051a7225424ebb64aedb21bb97df (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:20.853783 10679 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:20.854833 10743 ts_tablet_manager.cc:1929] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:20.855288 10743 log.cc:1199] T 02a63423ae464eb7b415a38419524150 P 27f97a330f374849be58a168318c1ec6: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root/wals/02a63423ae464eb7b415a38419524150
I20260430 07:58:20.856894 9748 catalog_manager.cc:5002] TS 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): tablet 02a63423ae464eb7b415a38419524150 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:20.861716 9858 raft_consensus.cc:2955] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.861764 10748 raft_consensus.cc:2955] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.863478 10680 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:20.875044 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet b978c5a9ab41402c806352f5ace14295 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:20.883015 9761 catalog_manager.cc:5671] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:20.887598 10718 raft_consensus.cc:2955] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.894563 10002 tablet_service.cc:1558] Processing DeleteTablet for tablet b978c5a9ab41402c806352f5ace14295 with delete_type TABLET_DATA_TOMBSTONED (TS 48e81934d52a4e93be5171ceea193032 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44006
I20260430 07:58:20.901523 10004 raft_consensus.cc:2955] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.902005 10567 raft_consensus.cc:2955] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.903767 10566 raft_consensus.cc:2955] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.927331 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 3028e67ff7d948ee8accee91fbcb70c3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:20.927965 10749 tablet_replica.cc:333] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032: stopping tablet replica
I20260430 07:58:20.929502 10749 raft_consensus.cc:2243] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:20.931222 9761 catalog_manager.cc:5671] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:20.943171 10749 raft_consensus.cc:2272] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:20.950289 10749 ts_tablet_manager.cc:1916] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:20.961360 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet 3028e67ff7d948ee8accee91fbcb70c3 with delete_type TABLET_DATA_TOMBSTONED (TS 27f97a330f374849be58a168318c1ec6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37818
I20260430 07:58:20.962383 10743 tablet_replica.cc:333] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6: stopping tablet replica
I20260430 07:58:20.963403 10743 raft_consensus.cc:2243] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:20.964126 10743 raft_consensus.cc:2272] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:20.969589 10743 ts_tablet_manager.cc:1916] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:20.987289 10749 ts_tablet_manager.cc:1929] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:20.987739 10749 log.cc:1199] T b978c5a9ab41402c806352f5ace14295 P 48e81934d52a4e93be5171ceea193032: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-2-root/wals/b978c5a9ab41402c806352f5ace14295
I20260430 07:58:20.989742 9747 catalog_manager.cc:5002] TS 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): tablet b978c5a9ab41402c806352f5ace14295 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:20.995291 10743 ts_tablet_manager.cc:1929] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:20.995751 10743 log.cc:1199] T 3028e67ff7d948ee8accee91fbcb70c3 P 27f97a330f374849be58a168318c1ec6: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root/wals/3028e67ff7d948ee8accee91fbcb70c3
I20260430 07:58:20.998351 9748 catalog_manager.cc:5002] TS 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): tablet 3028e67ff7d948ee8accee91fbcb70c3 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:22.064843 9937 consensus_queue.cc:237] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:22.075222 10010 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260430 07:58:22.075420 9903 consensus_peers.cc:597] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c393abc9c604fc9aff3cedf85f68725. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:22.077257 10718 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:22.081602 9858 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:22.083123 10718 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:22.089761 10680 raft_consensus.cc:2955] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.093406 10010 raft_consensus.cc:2955] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.105611 9860 consensus_queue.cc:237] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:22.111048 9758 catalog_manager.cc:5671] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:22.101173 9858 raft_consensus.cc:2955] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.156999 10010 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260430 07:58:22.160334 9828 consensus_peers.cc:597] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e51196f9c4fb44bf8f6296f5394417bb. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:22.161132 10751 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:22.171519 9937 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:22.185830 10746 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:22.258395 10714 raft_consensus.cc:2955] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.264818 9935 raft_consensus.cc:2955] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.267524 10008 raft_consensus.cc:2955] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.296432 9935 consensus_queue.cc:237] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:22.299507 9759 catalog_manager.cc:5671] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.312968 10008 raft_consensus.cc:1275] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:22.314343 9858 raft_consensus.cc:1275] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:22.315622 10750 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260430 07:58:22.318414 9903 consensus_peers.cc:597] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e537f351c19b43fd8b0b0a89cf897487. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:22.327900 10718 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:22.330482 10680 raft_consensus.cc:2955] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.336107 10008 raft_consensus.cc:2955] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.360350 9935 consensus_queue.cc:237] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:22.372720 9759 catalog_manager.cc:5671] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.389475 9863 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:22.387995 9860 raft_consensus.cc:2955] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
W20260430 07:58:22.392722 9903 consensus_peers.cc:597] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f111c6c48d34a6fb38048804967ed93. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:22.393518 10680 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:22.394910 10008 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:22.400334 10750 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:22.430374 10769 raft_consensus.cc:2955] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.432811 9863 raft_consensus.cc:2955] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.455786 10010 raft_consensus.cc:2955] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.463884 9761 catalog_manager.cc:5671] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:22.674432 10772 ts_tablet_manager.cc:933] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:22.691979 10772 tablet_copy_client.cc:323] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.130:38147
I20260430 07:58:22.695596 9948 tablet_copy_service.cc:140] P 77e19ab653f540339c1c613fad1451b5: Received BeginTabletCopySession request for tablet 7c393abc9c604fc9aff3cedf85f68725 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:58726)
I20260430 07:58:22.696270 9948 tablet_copy_service.cc:161] P 77e19ab653f540339c1c613fad1451b5: Beginning new tablet copy session on tablet 7c393abc9c604fc9aff3cedf85f68725 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:58726: session id = d66db415f0f0402fa431349afbf59888-7c393abc9c604fc9aff3cedf85f68725
I20260430 07:58:22.716506 9948 tablet_copy_source_session.cc:215] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:22.723752 10772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c393abc9c604fc9aff3cedf85f68725. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:22.740180 10774 ts_tablet_manager.cc:933] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679)
I20260430 07:58:22.767019 10774 tablet_copy_client.cc:323] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.129:36679
I20260430 07:58:22.769181 9873 tablet_copy_service.cc:140] P 27f97a330f374849be58a168318c1ec6: Received BeginTabletCopySession request for tablet e51196f9c4fb44bf8f6296f5394417bb from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:41094)
I20260430 07:58:22.768949 10772 tablet_copy_client.cc:806] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:22.769819 9873 tablet_copy_service.cc:161] P 27f97a330f374849be58a168318c1ec6: Beginning new tablet copy session on tablet e51196f9c4fb44bf8f6296f5394417bb from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:41094: session id = d66db415f0f0402fa431349afbf59888-e51196f9c4fb44bf8f6296f5394417bb
I20260430 07:58:22.770294 10772 tablet_copy_client.cc:670] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:22.776000 10772 tablet_copy_client.cc:538] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:22.782668 9873 tablet_copy_source_session.cc:215] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:22.786976 10774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e51196f9c4fb44bf8f6296f5394417bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:22.793085 10772 tablet_bootstrap.cc:492] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:22.809051 10777 ts_tablet_manager.cc:933] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:22.818394 10777 tablet_copy_client.cc:323] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.130:38147
I20260430 07:58:22.824014 9948 tablet_copy_service.cc:140] P 77e19ab653f540339c1c613fad1451b5: Received BeginTabletCopySession request for tablet e537f351c19b43fd8b0b0a89cf897487 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:58726)
I20260430 07:58:22.824621 9948 tablet_copy_service.cc:161] P 77e19ab653f540339c1c613fad1451b5: Beginning new tablet copy session on tablet e537f351c19b43fd8b0b0a89cf897487 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:58726: session id = d66db415f0f0402fa431349afbf59888-e537f351c19b43fd8b0b0a89cf897487
I20260430 07:58:22.825212 10774 tablet_copy_client.cc:806] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:22.825814 10774 tablet_copy_client.cc:670] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:22.835451 10774 tablet_copy_client.cc:538] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:22.848274 9948 tablet_copy_source_session.cc:215] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:22.849028 10774 tablet_bootstrap.cc:492] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:22.856455 10777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e537f351c19b43fd8b0b0a89cf897487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:22.887532 10772 tablet_bootstrap.cc:492] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:22.888633 10772 tablet_bootstrap.cc:492] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:22.889463 10772 ts_tablet_manager.cc:1403] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.097s user 0.024s sys 0.012s
I20260430 07:58:22.890265 10777 tablet_copy_client.cc:806] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:22.890939 10777 tablet_copy_client.cc:670] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:22.893154 10772 raft_consensus.cc:359] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:22.894939 10772 raft_consensus.cc:740] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:22.896589 10772 consensus_queue.cc:260] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:22.899030 10777 tablet_copy_client.cc:538] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:22.904245 10774 tablet_bootstrap.cc:492] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:22.906023 10774 tablet_bootstrap.cc:492] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:22.908576 10774 ts_tablet_manager.cc:1403] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.060s user 0.020s sys 0.004s
I20260430 07:58:22.911998 10774 raft_consensus.cc:359] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:22.918709 10777 tablet_bootstrap.cc:492] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:22.918908 10774 raft_consensus.cc:740] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:22.918625 10772 ts_tablet_manager.cc:1434] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.029s user 0.007s sys 0.003s
I20260430 07:58:22.919936 10774 consensus_queue.cc:260] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:22.922403 9948 tablet_copy_service.cc:342] P 77e19ab653f540339c1c613fad1451b5: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-7c393abc9c604fc9aff3cedf85f68725 received from {username='slave'} at 127.0.0.1:58726
I20260430 07:58:22.922962 9948 tablet_copy_service.cc:434] P 77e19ab653f540339c1c613fad1451b5: ending tablet copy session d66db415f0f0402fa431349afbf59888-7c393abc9c604fc9aff3cedf85f68725 on tablet 7c393abc9c604fc9aff3cedf85f68725 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:22.924374 10774 ts_tablet_manager.cc:1434] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260430 07:58:22.934517 9873 tablet_copy_service.cc:342] P 27f97a330f374849be58a168318c1ec6: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-e51196f9c4fb44bf8f6296f5394417bb received from {username='slave'} at 127.0.0.1:41094
I20260430 07:58:22.935071 9873 tablet_copy_service.cc:434] P 27f97a330f374849be58a168318c1ec6: ending tablet copy session d66db415f0f0402fa431349afbf59888-e51196f9c4fb44bf8f6296f5394417bb on tablet e51196f9c4fb44bf8f6296f5394417bb with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:22.954898 10777 tablet_bootstrap.cc:492] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:22.955874 10777 tablet_bootstrap.cc:492] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:22.956694 10777 ts_tablet_manager.cc:1403] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.038s user 0.024s sys 0.005s
I20260430 07:58:22.959383 10777 raft_consensus.cc:359] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:22.960208 10777 raft_consensus.cc:740] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:22.960856 10777 consensus_queue.cc:260] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:22.962963 10777 ts_tablet_manager.cc:1434] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260430 07:58:22.964500 9948 tablet_copy_service.cc:342] P 77e19ab653f540339c1c613fad1451b5: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-e537f351c19b43fd8b0b0a89cf897487 received from {username='slave'} at 127.0.0.1:58726
I20260430 07:58:22.964934 9948 tablet_copy_service.cc:434] P 77e19ab653f540339c1c613fad1451b5: ending tablet copy session d66db415f0f0402fa431349afbf59888-e537f351c19b43fd8b0b0a89cf897487 on tablet e537f351c19b43fd8b0b0a89cf897487 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:23.024206 10777 ts_tablet_manager.cc:933] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:23.027498 10777 tablet_copy_client.cc:323] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.130:38147
I20260430 07:58:23.029407 9948 tablet_copy_service.cc:140] P 77e19ab653f540339c1c613fad1451b5: Received BeginTabletCopySession request for tablet 7f111c6c48d34a6fb38048804967ed93 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:58726)
I20260430 07:58:23.029956 9948 tablet_copy_service.cc:161] P 77e19ab653f540339c1c613fad1451b5: Beginning new tablet copy session on tablet 7f111c6c48d34a6fb38048804967ed93 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:58726: session id = d66db415f0f0402fa431349afbf59888-7f111c6c48d34a6fb38048804967ed93
I20260430 07:58:23.038594 9948 tablet_copy_source_session.cc:215] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:23.042128 10777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f111c6c48d34a6fb38048804967ed93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:23.065532 10777 tablet_copy_client.cc:806] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:23.066190 10777 tablet_copy_client.cc:670] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:23.070379 10777 tablet_copy_client.cc:538] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:23.084168 10777 tablet_bootstrap.cc:492] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:23.110381 10777 tablet_bootstrap.cc:492] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:23.111395 10777 tablet_bootstrap.cc:492] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:23.112154 10777 ts_tablet_manager.cc:1403] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.012s
I20260430 07:58:23.115306 10777 raft_consensus.cc:359] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:23.116552 10777 raft_consensus.cc:740] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:23.117645 10777 consensus_queue.cc:260] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:23.175738 10777 ts_tablet_manager.cc:1434] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.063s user 0.011s sys 0.031s
I20260430 07:58:23.178040 9948 tablet_copy_service.cc:342] P 77e19ab653f540339c1c613fad1451b5: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-7f111c6c48d34a6fb38048804967ed93 received from {username='slave'} at 127.0.0.1:58726
I20260430 07:58:23.179019 9948 tablet_copy_service.cc:434] P 77e19ab653f540339c1c613fad1451b5: ending tablet copy session d66db415f0f0402fa431349afbf59888-7f111c6c48d34a6fb38048804967ed93 on tablet 7f111c6c48d34a6fb38048804967ed93 with peer d66db415f0f0402fa431349afbf59888
W20260430 07:58:23.258962 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 45487.3us
I20260430 07:58:23.280397 10566 raft_consensus.cc:1217] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:23.282197 10568 raft_consensus.cc:1217] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260430 07:58:23.339695 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 45286.5us
I20260430 07:58:23.401531 10568 raft_consensus.cc:1217] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:23.504091 10568 raft_consensus.cc:1217] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:23.645884 10792 raft_consensus.cc:1064] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:23.648559 10792 consensus_queue.cc:237] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:23.662387 10568 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:23.665587 10791 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:23.672497 10008 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:23.674932 10790 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:23.683329 9860 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:23.690119 10791 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:23.713048 10786 raft_consensus.cc:1064] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:23.716593 10786 consensus_queue.cc:237] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:23.706636 10791 raft_consensus.cc:2955] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.723955 9863 raft_consensus.cc:2955] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.712582 10568 raft_consensus.cc:2955] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.744453 10008 raft_consensus.cc:2955] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.749840 10010 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:23.749897 10566 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:23.751796 10788 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:23.753469 10787 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:23.756819 9758 catalog_manager.cc:5671] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:23.781400 10789 raft_consensus.cc:1064] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:23.787272 9933 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:23.790419 10787 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:23.798111 10786 raft_consensus.cc:2955] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.806435 10010 raft_consensus.cc:2955] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.805033 10789 consensus_queue.cc:237] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:23.836194 9938 raft_consensus.cc:2955] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.854609 10566 raft_consensus.cc:1275] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:23.860234 10790 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:23.865353 10008 raft_consensus.cc:1275] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:23.867326 10789 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:23.868260 9860 raft_consensus.cc:1275] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:23.871068 10790 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:23.887876 10568 raft_consensus.cc:2955] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.903123 9935 consensus_queue.cc:237] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:23.910833 10789 raft_consensus.cc:2955] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.913596 10565 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:23.919622 9761 catalog_manager.cc:5671] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.923625 10768 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:23.929814 10791 raft_consensus.cc:1064] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:23.933801 10565 raft_consensus.cc:2955] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.939627 10790 raft_consensus.cc:2955] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.953989 10791 consensus_queue.cc:237] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:23.919826 9860 raft_consensus.cc:2955] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.962407 9863 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:23.968592 9862 consensus_queue.cc:237] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:23.989197 10566 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:23.990772 10003 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:23.993412 10787 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:23.993422 9761 catalog_manager.cc:5671] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:23.995997 10792 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:23.998996 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 7c393abc9c604fc9aff3cedf85f68725 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:24.004228 10785 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260430 07:58:24.009924 10010 raft_consensus.cc:2955] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.029538 9862 raft_consensus.cc:2955] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.041132 10010 raft_consensus.cc:2955] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.038626 10786 raft_consensus.cc:2955] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.045210 10568 raft_consensus.cc:2955] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.054196 10566 raft_consensus.cc:2955] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.057781 10002 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c393abc9c604fc9aff3cedf85f68725 with delete_type TABLET_DATA_TOMBSTONED (TS 48e81934d52a4e93be5171ceea193032 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44006
I20260430 07:58:24.058739 9760 catalog_manager.cc:5671] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.092934 10008 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:24.094617 10568 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:24.108891 10790 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:24.112879 10815 tablet_replica.cc:333] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032: stopping tablet replica
I20260430 07:58:24.112102 9759 catalog_manager.cc:5671] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.115485 10815 raft_consensus.cc:2243] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:24.117667 10790 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:24.118687 10815 raft_consensus.cc:2272] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:24.127151 9862 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:24.128995 9748 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet e51196f9c4fb44bf8f6296f5394417bb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:24.130090 10791 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:24.152263 10815 ts_tablet_manager.cc:1916] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:24.164345 9924 tablet_service.cc:1558] Processing DeleteTablet for tablet e51196f9c4fb44bf8f6296f5394417bb with delete_type TABLET_DATA_TOMBSTONED (TS 77e19ab653f540339c1c613fad1451b5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35252
I20260430 07:58:24.166945 9935 consensus_queue.cc:237] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:24.167069 10813 raft_consensus.cc:2955] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.177165 10820 tablet_replica.cc:333] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5: stopping tablet replica
I20260430 07:58:24.189812 10820 raft_consensus.cc:2243] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:24.190685 10820 raft_consensus.cc:2272] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:24.192260 10815 ts_tablet_manager.cc:1929] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:24.192714 10815 log.cc:1199] T 7c393abc9c604fc9aff3cedf85f68725 P 48e81934d52a4e93be5171ceea193032: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-2-root/wals/7c393abc9c604fc9aff3cedf85f68725
I20260430 07:58:24.192845 10568 raft_consensus.cc:2955] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.200139 9747 catalog_manager.cc:5002] TS 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): tablet 7c393abc9c604fc9aff3cedf85f68725 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:24.212569 9858 raft_consensus.cc:1275] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:24.213343 10566 raft_consensus.cc:1275] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:24.217438 10010 raft_consensus.cc:2955] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.220222 9759 catalog_manager.cc:5671] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.218595 10813 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:24.226595 10814 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:24.233186 9861 raft_consensus.cc:2955] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.277336 10820 ts_tablet_manager.cc:1916] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:24.288251 10789 raft_consensus.cc:2955] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.309326 9862 raft_consensus.cc:2955] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.310734 9935 consensus_queue.cc:237] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:24.305702 10568 raft_consensus.cc:2955] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.328434 10565 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:24.330580 10010 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:24.332170 10813 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:24.348520 10813 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:24.355192 10811 raft_consensus.cc:2955] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.356392 10820 ts_tablet_manager.cc:1929] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:24.357129 10820 log.cc:1199] T e51196f9c4fb44bf8f6296f5394417bb P 77e19ab653f540339c1c613fad1451b5: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-1-root/wals/e51196f9c4fb44bf8f6296f5394417bb
I20260430 07:58:24.357422 9759 catalog_manager.cc:5671] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.366786 9747 catalog_manager.cc:5002] TS 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): tablet e51196f9c4fb44bf8f6296f5394417bb (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:24.408187 10568 raft_consensus.cc:2955] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.420970 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet e537f351c19b43fd8b0b0a89cf897487 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:24.427523 10010 raft_consensus.cc:2955] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:24.437515 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 7f111c6c48d34a6fb38048804967ed93 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:24.458775 9761 catalog_manager.cc:5671] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:24.467589 10002 tablet_service.cc:1558] Processing DeleteTablet for tablet e537f351c19b43fd8b0b0a89cf897487 with delete_type TABLET_DATA_TOMBSTONED (TS 48e81934d52a4e93be5171ceea193032 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44006
I20260430 07:58:24.468779 10815 tablet_replica.cc:333] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032: stopping tablet replica
I20260430 07:58:24.469944 10815 raft_consensus.cc:2243] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:24.470638 10815 raft_consensus.cc:2272] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:24.475881 10815 ts_tablet_manager.cc:1916] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:24.493880 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f111c6c48d34a6fb38048804967ed93 with delete_type TABLET_DATA_TOMBSTONED (TS 27f97a330f374849be58a168318c1ec6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37818
I20260430 07:58:24.501884 10815 ts_tablet_manager.cc:1929] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:24.503687 10815 log.cc:1199] T e537f351c19b43fd8b0b0a89cf897487 P 48e81934d52a4e93be5171ceea193032: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-2-root/wals/e537f351c19b43fd8b0b0a89cf897487
I20260430 07:58:24.506296 10822 tablet_replica.cc:333] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6: stopping tablet replica
I20260430 07:58:24.507100 9747 catalog_manager.cc:5002] TS 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): tablet e537f351c19b43fd8b0b0a89cf897487 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:24.507575 10822 raft_consensus.cc:2243] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:24.508545 10822 raft_consensus.cc:2272] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:24.511487 10822 ts_tablet_manager.cc:1916] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:24.527201 10822 ts_tablet_manager.cc:1929] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:24.527801 10822 log.cc:1199] T 7f111c6c48d34a6fb38048804967ed93 P 27f97a330f374849be58a168318c1ec6: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root/wals/7f111c6c48d34a6fb38048804967ed93
I20260430 07:58:24.529387 9748 catalog_manager.cc:5002] TS 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): tablet 7f111c6c48d34a6fb38048804967ed93 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:25.561746 10010 consensus_queue.cc:237] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:25.569550 9860 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260430 07:58:25.570639 9977 consensus_peers.cc:597] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de66bf2ed62e49aa95b1deff0383d037. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:25.570900 9935 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:25.570900 10794 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:25.573570 10821 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:25.581979 10821 raft_consensus.cc:2955] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.585204 9858 raft_consensus.cc:2955] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.586440 9935 raft_consensus.cc:2955] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.601022 9933 consensus_queue.cc:237] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:25.603170 9761 catalog_manager.cc:5671] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260430 07:58:25.617146 9903 consensus_peers.cc:597] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 202cb0ccf6a947e6bdcffc383288322c. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:25.620905 10010 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:25.621984 9862 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:25.625694 10789 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:25.626793 10813 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:25.639603 10813 raft_consensus.cc:2955] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.642396 9862 raft_consensus.cc:2955] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.667879 9758 catalog_manager.cc:5671] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.669139 9933 consensus_queue.cc:237] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:25.673954 10010 raft_consensus.cc:2955] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
W20260430 07:58:25.699203 9903 consensus_peers.cc:597] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58ea2e276db24da3890f352cef28ff97. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:25.699604 10003 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:25.701076 10814 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:25.707827 9862 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:25.709360 10814 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:25.732151 10813 raft_consensus.cc:2955] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.733636 10010 raft_consensus.cc:2955] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.741500 9862 raft_consensus.cc:2955] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.749118 9933 consensus_queue.cc:237] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:25.757122 9758 catalog_manager.cc:5671] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:25.764870 10010 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260430 07:58:25.766647 9903 consensus_peers.cc:597] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db24b74e2fa44ae2beba675582424a3b. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:25.767020 9862 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:25.768127 10813 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:25.769014 10836 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:25.787390 10836 raft_consensus.cc:2955] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.795054 10010 raft_consensus.cc:2955] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.798429 9862 raft_consensus.cc:2955] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:25.817415 9760 catalog_manager.cc:5671] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:26.062639 10841 ts_tablet_manager.cc:933] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:26.065788 10841 tablet_copy_client.cc:323] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.130:38147
I20260430 07:58:26.067612 9948 tablet_copy_service.cc:140] P 77e19ab653f540339c1c613fad1451b5: Received BeginTabletCopySession request for tablet 202cb0ccf6a947e6bdcffc383288322c from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:58726)
I20260430 07:58:26.068238 9948 tablet_copy_service.cc:161] P 77e19ab653f540339c1c613fad1451b5: Beginning new tablet copy session on tablet 202cb0ccf6a947e6bdcffc383288322c from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:58726: session id = d66db415f0f0402fa431349afbf59888-202cb0ccf6a947e6bdcffc383288322c
I20260430 07:58:26.076153 9948 tablet_copy_source_session.cc:215] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:26.080003 10841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 202cb0ccf6a947e6bdcffc383288322c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:26.096330 10841 tablet_copy_client.cc:806] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:26.096912 10841 tablet_copy_client.cc:670] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:26.101425 10841 tablet_copy_client.cc:538] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:26.115532 10841 tablet_bootstrap.cc:492] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:26.125862 10843 ts_tablet_manager.cc:933] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:26.130951 10843 tablet_copy_client.cc:323] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.131:39205
I20260430 07:58:26.133047 10022 tablet_copy_service.cc:140] P 48e81934d52a4e93be5171ceea193032: Received BeginTabletCopySession request for tablet de66bf2ed62e49aa95b1deff0383d037 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:53754)
I20260430 07:58:26.133814 10022 tablet_copy_service.cc:161] P 48e81934d52a4e93be5171ceea193032: Beginning new tablet copy session on tablet de66bf2ed62e49aa95b1deff0383d037 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:53754: session id = d66db415f0f0402fa431349afbf59888-de66bf2ed62e49aa95b1deff0383d037
I20260430 07:58:26.148026 10841 tablet_bootstrap.cc:492] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:26.149554 10841 tablet_bootstrap.cc:492] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:26.150616 10841 ts_tablet_manager.cc:1403] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.013s
I20260430 07:58:26.153622 10841 raft_consensus.cc:359] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:26.154480 10841 raft_consensus.cc:740] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:26.155521 10841 consensus_queue.cc:260] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:26.163594 10022 tablet_copy_source_session.cc:215] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:26.168881 10843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de66bf2ed62e49aa95b1deff0383d037. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:26.184092 10841 ts_tablet_manager.cc:1434] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.033s user 0.012s sys 0.020s
I20260430 07:58:26.186079 10847 ts_tablet_manager.cc:933] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:26.186947 9948 tablet_copy_service.cc:342] P 77e19ab653f540339c1c613fad1451b5: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-202cb0ccf6a947e6bdcffc383288322c received from {username='slave'} at 127.0.0.1:58726
I20260430 07:58:26.187371 9948 tablet_copy_service.cc:434] P 77e19ab653f540339c1c613fad1451b5: ending tablet copy session d66db415f0f0402fa431349afbf59888-202cb0ccf6a947e6bdcffc383288322c on tablet 202cb0ccf6a947e6bdcffc383288322c with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:26.194602 10847 tablet_copy_client.cc:323] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.130:38147
I20260430 07:58:26.197980 10843 tablet_copy_client.cc:806] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:26.199709 10843 tablet_copy_client.cc:670] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:26.201006 9948 tablet_copy_service.cc:140] P 77e19ab653f540339c1c613fad1451b5: Received BeginTabletCopySession request for tablet db24b74e2fa44ae2beba675582424a3b from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:58726)
I20260430 07:58:26.201632 9948 tablet_copy_service.cc:161] P 77e19ab653f540339c1c613fad1451b5: Beginning new tablet copy session on tablet db24b74e2fa44ae2beba675582424a3b from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:58726: session id = d66db415f0f0402fa431349afbf59888-db24b74e2fa44ae2beba675582424a3b
I20260430 07:58:26.203632 10843 tablet_copy_client.cc:538] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:26.213169 10843 tablet_bootstrap.cc:492] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:26.223762 9948 tablet_copy_source_session.cc:215] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:26.227365 10847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db24b74e2fa44ae2beba675582424a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:26.236155 10843 tablet_bootstrap.cc:492] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:26.237125 10843 tablet_bootstrap.cc:492] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:26.238004 10843 ts_tablet_manager.cc:1403] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.025s user 0.021s sys 0.005s
I20260430 07:58:26.240912 10843 raft_consensus.cc:359] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:26.241739 10843 raft_consensus.cc:740] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:26.242365 10843 consensus_queue.cc:260] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:26.246428 10843 ts_tablet_manager.cc:1434] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260430 07:58:26.248118 10022 tablet_copy_service.cc:342] P 48e81934d52a4e93be5171ceea193032: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-de66bf2ed62e49aa95b1deff0383d037 received from {username='slave'} at 127.0.0.1:53754
I20260430 07:58:26.248564 10022 tablet_copy_service.cc:434] P 48e81934d52a4e93be5171ceea193032: ending tablet copy session d66db415f0f0402fa431349afbf59888-de66bf2ed62e49aa95b1deff0383d037 on tablet de66bf2ed62e49aa95b1deff0383d037 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:26.248934 10847 tablet_copy_client.cc:806] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:26.249825 10847 tablet_copy_client.cc:670] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:26.254210 10847 tablet_copy_client.cc:538] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:26.266959 10847 tablet_bootstrap.cc:492] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:26.305802 10843 ts_tablet_manager.cc:933] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:26.308650 10843 tablet_copy_client.cc:323] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.130:38147
I20260430 07:58:26.310216 9948 tablet_copy_service.cc:140] P 77e19ab653f540339c1c613fad1451b5: Received BeginTabletCopySession request for tablet 58ea2e276db24da3890f352cef28ff97 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:58726)
I20260430 07:58:26.310745 9948 tablet_copy_service.cc:161] P 77e19ab653f540339c1c613fad1451b5: Beginning new tablet copy session on tablet 58ea2e276db24da3890f352cef28ff97 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:58726: session id = d66db415f0f0402fa431349afbf59888-58ea2e276db24da3890f352cef28ff97
I20260430 07:58:26.314831 10847 tablet_bootstrap.cc:492] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:26.315773 10847 tablet_bootstrap.cc:492] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:26.316476 10847 ts_tablet_manager.cc:1403] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.050s user 0.026s sys 0.016s
I20260430 07:58:26.318034 9948 tablet_copy_source_session.cc:215] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:26.319710 10847 raft_consensus.cc:359] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:26.320564 10847 raft_consensus.cc:740] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:26.321223 10847 consensus_queue.cc:260] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:26.321997 10843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ea2e276db24da3890f352cef28ff97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:26.323699 10847 ts_tablet_manager.cc:1434] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260430 07:58:26.325582 9948 tablet_copy_service.cc:342] P 77e19ab653f540339c1c613fad1451b5: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-db24b74e2fa44ae2beba675582424a3b received from {username='slave'} at 127.0.0.1:58726
I20260430 07:58:26.326050 9948 tablet_copy_service.cc:434] P 77e19ab653f540339c1c613fad1451b5: ending tablet copy session d66db415f0f0402fa431349afbf59888-db24b74e2fa44ae2beba675582424a3b on tablet db24b74e2fa44ae2beba675582424a3b with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:26.349766 10843 tablet_copy_client.cc:806] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:26.350353 10843 tablet_copy_client.cc:670] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:26.356257 10843 tablet_copy_client.cc:538] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:26.367261 10843 tablet_bootstrap.cc:492] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:26.387709 10843 tablet_bootstrap.cc:492] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:26.388566 10843 tablet_bootstrap.cc:492] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:26.389148 10843 ts_tablet_manager.cc:1403] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.005s
I20260430 07:58:26.392050 10843 raft_consensus.cc:359] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:26.392689 10843 raft_consensus.cc:740] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:26.393209 10843 consensus_queue.cc:260] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:26.396015 10843 ts_tablet_manager.cc:1434] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260430 07:58:26.397598 9948 tablet_copy_service.cc:342] P 77e19ab653f540339c1c613fad1451b5: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-58ea2e276db24da3890f352cef28ff97 received from {username='slave'} at 127.0.0.1:58726
I20260430 07:58:26.398437 9948 tablet_copy_service.cc:434] P 77e19ab653f540339c1c613fad1451b5: ending tablet copy session d66db415f0f0402fa431349afbf59888-58ea2e276db24da3890f352cef28ff97 on tablet 58ea2e276db24da3890f352cef28ff97 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:26.543156 10565 raft_consensus.cc:1217] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:26.605024 10565 raft_consensus.cc:1217] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:26.699137 10565 raft_consensus.cc:1217] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:26.817423 10565 raft_consensus.cc:1217] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:26.969252 10821 raft_consensus.cc:1064] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:26.971771 10821 consensus_queue.cc:237] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:26.978307 9860 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:26.978056 9935 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:26.979001 10565 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:26.980012 10721 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:26.982306 10721 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:26.987237 10794 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:26.990885 10836 raft_consensus.cc:1064] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:27.000782 10836 consensus_queue.cc:237] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:27.021251 10721 raft_consensus.cc:2955] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.023192 10565 raft_consensus.cc:2955] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.037250 9858 raft_consensus.cc:2955] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.044697 9860 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:27.046249 10565 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:27.046823 9761 catalog_manager.cc:5671] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.049935 10789 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:27.054694 10003 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:27.056160 10814 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:27.057062 10813 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:27.058573 9935 raft_consensus.cc:2955] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.083097 10789 raft_consensus.cc:2955] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.085480 10565 raft_consensus.cc:2955] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.089391 9858 raft_consensus.cc:2955] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.130991 10003 raft_consensus.cc:2955] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.165596 10003 consensus_queue.cc:237] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:27.179201 10565 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:27.181214 9858 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:27.182220 9760 catalog_manager.cc:5671] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:27.184616 10821 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:27.186466 10854 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:27.200516 9861 raft_consensus.cc:2955] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.195713 10794 raft_consensus.cc:2955] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.222755 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet de66bf2ed62e49aa95b1deff0383d037 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:27.237092 10813 raft_consensus.cc:1064] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:27.239517 9758 catalog_manager.cc:5671] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 reported cstate change: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.245956 10565 raft_consensus.cc:2955] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.240420 10813 consensus_queue.cc:237] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:27.260739 10568 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:27.260739 10008 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:27.262285 9924 tablet_service.cc:1558] Processing DeleteTablet for tablet de66bf2ed62e49aa95b1deff0383d037 with delete_type TABLET_DATA_TOMBSTONED (TS 77e19ab653f540339c1c613fad1451b5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35252
I20260430 07:58:27.264259 9938 consensus_queue.cc:237] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:27.264954 9861 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:27.267382 10869 tablet_replica.cc:333] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5: stopping tablet replica
I20260430 07:58:27.267118 10836 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:27.268332 10869 raft_consensus.cc:2243] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:27.268870 10869 raft_consensus.cc:2272] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:27.271175 10814 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:27.273519 10789 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:27.296159 10565 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:27.297935 10836 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260430 07:58:27.306385 10869 ts_tablet_manager.cc:1916] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:27.313926 9858 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:27.313657 10813 raft_consensus.cc:2955] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.317405 10814 raft_consensus.cc:1064] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:27.322546 10814 consensus_queue.cc:237] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:27.323390 10010 raft_consensus.cc:2955] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.328013 9858 raft_consensus.cc:2955] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.342950 10869 ts_tablet_manager.cc:1929] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:27.344192 10869 log.cc:1199] T de66bf2ed62e49aa95b1deff0383d037 P 77e19ab653f540339c1c613fad1451b5: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-1-root/wals/de66bf2ed62e49aa95b1deff0383d037
I20260430 07:58:27.348456 10836 raft_consensus.cc:2955] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.350349 10565 raft_consensus.cc:2955] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.342682 10568 raft_consensus.cc:2955] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.355755 9747 catalog_manager.cc:5002] TS 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): tablet de66bf2ed62e49aa95b1deff0383d037 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:27.352568 9760 catalog_manager.cc:5671] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.354264 10864 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260430 07:58:27.373162 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 202cb0ccf6a947e6bdcffc383288322c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:27.384282 9759 catalog_manager.cc:5671] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.388721 10010 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:27.390396 9858 raft_consensus.cc:2955] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 48e81934d52a4e93be5171ceea193032 (127.8.119.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.396373 10814 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260430 07:58:27.401454 9862 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:27.411765 10568 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:27.418848 10002 tablet_service.cc:1558] Processing DeleteTablet for tablet 202cb0ccf6a947e6bdcffc383288322c with delete_type TABLET_DATA_TOMBSTONED (TS 48e81934d52a4e93be5171ceea193032 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44006
I20260430 07:58:27.424645 10789 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:27.427487 10789 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:27.439725 9933 consensus_queue.cc:237] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:27.441448 10876 tablet_replica.cc:333] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032: stopping tablet replica
I20260430 07:58:27.452410 10565 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:27.454809 10876 raft_consensus.cc:2243] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:27.455297 10010 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:27.454881 10874 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:27.458506 10814 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:27.458714 10876 raft_consensus.cc:2272] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:27.469008 10813 raft_consensus.cc:2955] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.473500 10836 raft_consensus.cc:2955] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.477893 10565 raft_consensus.cc:2955] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.482306 9862 raft_consensus.cc:2955] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.484189 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 58ea2e276db24da3890f352cef28ff97 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:27.482035 10010 raft_consensus.cc:2955] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.493763 10566 raft_consensus.cc:2955] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.496272 9759 catalog_manager.cc:5671] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:27.510445 10876 ts_tablet_manager.cc:1916] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:27.514010 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet 58ea2e276db24da3890f352cef28ff97 with delete_type TABLET_DATA_TOMBSTONED (TS 27f97a330f374849be58a168318c1ec6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37818
I20260430 07:58:27.519227 10010 raft_consensus.cc:2955] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.526714 9761 catalog_manager.cc:5671] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.545354 10876 ts_tablet_manager.cc:1929] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:27.548460 10876 log.cc:1199] T 202cb0ccf6a947e6bdcffc383288322c P 48e81934d52a4e93be5171ceea193032: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-2-root/wals/202cb0ccf6a947e6bdcffc383288322c
I20260430 07:58:27.550681 9747 catalog_manager.cc:5002] TS 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205): tablet 202cb0ccf6a947e6bdcffc383288322c (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:27.551586 10880 tablet_replica.cc:333] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6: stopping tablet replica
I20260430 07:58:27.552649 10880 raft_consensus.cc:2243] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:27.553529 10880 raft_consensus.cc:2272] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:27.559949 10880 ts_tablet_manager.cc:1916] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:27.564543 9938 consensus_queue.cc:237] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:27.574498 10566 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:27.574705 10010 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260430 07:58:27.580065 10874 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:27.581063 10880 ts_tablet_manager.cc:1929] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:27.581610 10880 log.cc:1199] T 58ea2e276db24da3890f352cef28ff97 P 27f97a330f374849be58a168318c1ec6: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root/wals/58ea2e276db24da3890f352cef28ff97
I20260430 07:58:27.581951 10874 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:27.583312 9748 catalog_manager.cc:5002] TS 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): tablet 58ea2e276db24da3890f352cef28ff97 (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:27.587740 10813 raft_consensus.cc:2955] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.593254 10010 raft_consensus.cc:2955] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.593633 10565 raft_consensus.cc:2955] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:27.596392 9747 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet db24b74e2fa44ae2beba675582424a3b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260430 07:58:27.610580 9759 catalog_manager.cc:5671] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index 3 to 4, VOTER 27f97a330f374849be58a168318c1ec6 (127.8.119.129) evicted. New cstate: current_term: 1 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:27.627635 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet db24b74e2fa44ae2beba675582424a3b with delete_type TABLET_DATA_TOMBSTONED (TS 27f97a330f374849be58a168318c1ec6 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37818
I20260430 07:58:27.628752 10880 tablet_replica.cc:333] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6: stopping tablet replica
I20260430 07:58:27.630930 10880 raft_consensus.cc:2243] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260430 07:58:27.631758 10880 raft_consensus.cc:2272] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260430 07:58:27.635818 10880 ts_tablet_manager.cc:1916] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260430 07:58:27.652796 10880 ts_tablet_manager.cc:1929] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260430 07:58:27.653117 10880 log.cc:1199] T db24b74e2fa44ae2beba675582424a3b P 27f97a330f374849be58a168318c1ec6: Deleting WAL directory at /tmp/dist-test-tasko4Lt3r/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1777535818658779-8670-0/minicluster-data/ts-0-root/wals/db24b74e2fa44ae2beba675582424a3b
I20260430 07:58:27.654590 9748 catalog_manager.cc:5002] TS 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679): tablet db24b74e2fa44ae2beba675582424a3b (table test-workload [id=cdb82ba68fb442a9ba38cbc332a24194]) successfully deleted
I20260430 07:58:28.712426 10010 consensus_queue.cc:237] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:28.720999 9938 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260430 07:58:28.722029 9977 consensus_peers.cc:597] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fe0ed01a03d4f93a1f8a9644c5497cf. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:28.722199 9862 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:28.723551 10721 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:28.724512 10794 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:28.733143 10794 raft_consensus.cc:2955] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.737690 9933 raft_consensus.cc:2955] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.740913 9862 raft_consensus.cc:2955] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.750808 9861 consensus_queue.cc:237] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:28.765473 9759 catalog_manager.cc:5671] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260430 07:58:28.768442 9828 consensus_peers.cc:597] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7303f2c98e1b411f83314a2c2f146d4b. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:28.769698 10010 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:28.772588 9938 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:28.774261 10786 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:28.776360 10786 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:28.801204 10893 raft_consensus.cc:2955] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.805372 10010 raft_consensus.cc:2955] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.809064 9933 raft_consensus.cc:2955] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.824316 10004 consensus_queue.cc:237] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:28.846897 9861 raft_consensus.cc:1275] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:28.847534 9933 raft_consensus.cc:1275] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:28.850594 10881 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:28.852053 10794 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260430 07:58:28.853466 9977 consensus_peers.cc:597] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 805814cfaf2b43ab87c90b2b7b56a211. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:28.854708 9761 catalog_manager.cc:5671] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.866194 10881 raft_consensus.cc:2955] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.868782 9938 raft_consensus.cc:2955] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.869118 9861 raft_consensus.cc:2955] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.882689 10004 consensus_queue.cc:237] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
W20260430 07:58:28.894740 9977 consensus_peers.cc:597] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 -> Peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723): Couldn't send request to peer d66db415f0f0402fa431349afbf59888. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8946312605d455cbfe6546ad6873ec1. This is attempt 1: this message will repeat every 5th retry.
I20260430 07:58:28.896715 9933 raft_consensus.cc:1275] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:28.900729 10721 consensus_queue.cc:1048] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:28.900261 9760 catalog_manager.cc:5671] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.902863 9862 raft_consensus.cc:1275] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260430 07:58:28.904673 10721 consensus_queue.cc:1048] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260430 07:58:28.916513 10794 raft_consensus.cc:2955] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.926527 9933 raft_consensus.cc:2955] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.928697 9862 raft_consensus.cc:2955] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } } }
I20260430 07:58:28.939113 9759 catalog_manager.cc:5671] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index -1 to 2, NON_VOTER d66db415f0f0402fa431349afbf59888 (127.8.119.132) added. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:29.293871 10903 ts_tablet_manager.cc:933] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679)
I20260430 07:58:29.299613 10903 tablet_copy_client.cc:323] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.129:36679
I20260430 07:58:29.302296 9873 tablet_copy_service.cc:140] P 27f97a330f374849be58a168318c1ec6: Received BeginTabletCopySession request for tablet 7303f2c98e1b411f83314a2c2f146d4b from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:41094)
I20260430 07:58:29.303237 9873 tablet_copy_service.cc:161] P 27f97a330f374849be58a168318c1ec6: Beginning new tablet copy session on tablet 7303f2c98e1b411f83314a2c2f146d4b from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:41094: session id = d66db415f0f0402fa431349afbf59888-7303f2c98e1b411f83314a2c2f146d4b
I20260430 07:58:29.310878 9873 tablet_copy_source_session.cc:215] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:29.314013 10903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7303f2c98e1b411f83314a2c2f146d4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:29.329393 10905 ts_tablet_manager.cc:933] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:29.331238 10903 tablet_copy_client.cc:806] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:29.332134 10903 tablet_copy_client.cc:670] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:29.352221 10903 tablet_copy_client.cc:538] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:29.352412 10905 tablet_copy_client.cc:323] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.131:39205
I20260430 07:58:29.357003 10022 tablet_copy_service.cc:140] P 48e81934d52a4e93be5171ceea193032: Received BeginTabletCopySession request for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:53754)
I20260430 07:58:29.358060 10022 tablet_copy_service.cc:161] P 48e81934d52a4e93be5171ceea193032: Beginning new tablet copy session on tablet 9fe0ed01a03d4f93a1f8a9644c5497cf from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:53754: session id = d66db415f0f0402fa431349afbf59888-9fe0ed01a03d4f93a1f8a9644c5497cf
I20260430 07:58:29.367566 10022 tablet_copy_source_session.cc:215] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:29.368371 10903 tablet_bootstrap.cc:492] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:29.371162 10905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:29.377121 10907 ts_tablet_manager.cc:933] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:29.388268 10907 tablet_copy_client.cc:323] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.131:39205
I20260430 07:58:29.391083 10022 tablet_copy_service.cc:140] P 48e81934d52a4e93be5171ceea193032: Received BeginTabletCopySession request for tablet 805814cfaf2b43ab87c90b2b7b56a211 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:53754)
I20260430 07:58:29.391937 10022 tablet_copy_service.cc:161] P 48e81934d52a4e93be5171ceea193032: Beginning new tablet copy session on tablet 805814cfaf2b43ab87c90b2b7b56a211 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:53754: session id = d66db415f0f0402fa431349afbf59888-805814cfaf2b43ab87c90b2b7b56a211
I20260430 07:58:29.392853 10903 tablet_bootstrap.cc:492] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:29.393836 10903 tablet_bootstrap.cc:492] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:29.394536 10903 ts_tablet_manager.cc:1403] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.028s user 0.025s sys 0.000s
I20260430 07:58:29.397657 10903 raft_consensus.cc:359] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:29.398496 10903 raft_consensus.cc:740] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:29.399133 10903 consensus_queue.cc:260] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:29.400959 10022 tablet_copy_source_session.cc:215] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:29.409199 10903 ts_tablet_manager.cc:1434] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260430 07:58:29.414175 9873 tablet_copy_service.cc:342] P 27f97a330f374849be58a168318c1ec6: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-7303f2c98e1b411f83314a2c2f146d4b received from {username='slave'} at 127.0.0.1:41094
I20260430 07:58:29.414690 9873 tablet_copy_service.cc:434] P 27f97a330f374849be58a168318c1ec6: ending tablet copy session d66db415f0f0402fa431349afbf59888-7303f2c98e1b411f83314a2c2f146d4b on tablet 7303f2c98e1b411f83314a2c2f146d4b with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:29.419023 10907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 805814cfaf2b43ab87c90b2b7b56a211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:29.420413 10910 ts_tablet_manager.cc:933] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: Initiating tablet copy from peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:29.456723 10905 tablet_copy_client.cc:806] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:29.457422 10905 tablet_copy_client.cc:670] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:29.465193 10910 tablet_copy_client.cc:323] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: tablet copy: Beginning tablet copy session from remote peer at address 127.8.119.131:39205
I20260430 07:58:29.467195 10905 tablet_copy_client.cc:538] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:29.467351 10022 tablet_copy_service.cc:140] P 48e81934d52a4e93be5171ceea193032: Received BeginTabletCopySession request for tablet f8946312605d455cbfe6546ad6873ec1 from peer d66db415f0f0402fa431349afbf59888 ({username='slave'} at 127.0.0.1:53754)
I20260430 07:58:29.468063 10022 tablet_copy_service.cc:161] P 48e81934d52a4e93be5171ceea193032: Beginning new tablet copy session on tablet f8946312605d455cbfe6546ad6873ec1 from peer d66db415f0f0402fa431349afbf59888 at {username='slave'} at 127.0.0.1:53754: session id = d66db415f0f0402fa431349afbf59888-f8946312605d455cbfe6546ad6873ec1
I20260430 07:58:29.476886 10022 tablet_copy_source_session.cc:215] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Tablet Copy: opened 0 blocks and 1 log segments
I20260430 07:58:29.480090 10910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8946312605d455cbfe6546ad6873ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260430 07:58:29.478806 10905 tablet_bootstrap.cc:492] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:29.491258 10907 tablet_copy_client.cc:806] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:29.491976 10907 tablet_copy_client.cc:670] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:29.496954 10907 tablet_copy_client.cc:538] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:29.514185 10910 tablet_copy_client.cc:806] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 0 data blocks...
I20260430 07:58:29.514889 10910 tablet_copy_client.cc:670] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: tablet copy: Starting download of 1 WAL segments...
I20260430 07:58:29.517227 10907 tablet_bootstrap.cc:492] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:29.527467 10905 tablet_bootstrap.cc:492] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:29.528492 10905 tablet_bootstrap.cc:492] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:29.529114 10910 tablet_copy_client.cc:538] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260430 07:58:29.529230 10905 ts_tablet_manager.cc:1403] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.051s user 0.019s sys 0.012s
I20260430 07:58:29.532456 10905 raft_consensus.cc:359] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:29.533347 10905 raft_consensus.cc:740] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:29.534041 10905 consensus_queue.cc:260] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:29.537139 10905 ts_tablet_manager.cc:1434] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260430 07:58:29.539634 10022 tablet_copy_service.cc:342] P 48e81934d52a4e93be5171ceea193032: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-9fe0ed01a03d4f93a1f8a9644c5497cf received from {username='slave'} at 127.0.0.1:53754
I20260430 07:58:29.540158 10022 tablet_copy_service.cc:434] P 48e81934d52a4e93be5171ceea193032: ending tablet copy session d66db415f0f0402fa431349afbf59888-9fe0ed01a03d4f93a1f8a9644c5497cf on tablet 9fe0ed01a03d4f93a1f8a9644c5497cf with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:29.548593 10910 tablet_bootstrap.cc:492] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: Bootstrap starting.
I20260430 07:58:29.585493 10907 tablet_bootstrap.cc:492] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:29.587620 10907 tablet_bootstrap.cc:492] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:29.588327 10907 ts_tablet_manager.cc:1403] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.071s user 0.025s sys 0.004s
I20260430 07:58:29.591284 10910 tablet_bootstrap.cc:492] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260430 07:58:29.591531 10907 raft_consensus.cc:359] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:29.592271 10910 tablet_bootstrap.cc:492] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: Bootstrap complete.
I20260430 07:58:29.592325 10907 raft_consensus.cc:740] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:29.593220 10910 ts_tablet_manager.cc:1403] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: Time spent bootstrapping tablet: real 0.045s user 0.022s sys 0.014s
I20260430 07:58:29.593238 10907 consensus_queue.cc:260] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:29.596912 10910 raft_consensus.cc:359] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:29.598037 10910 raft_consensus.cc:740] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Initialized, Role: LEARNER
I20260430 07:58:29.598714 10910 consensus_queue.cc:260] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:29.600968 10910 ts_tablet_manager.cc:1434] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20260430 07:58:29.606055 10907 ts_tablet_manager.cc:1434] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: Time spent starting tablet: real 0.017s user 0.009s sys 0.000s
I20260430 07:58:30.788847 10918 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:30.790627 10918 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:30.801905 10918 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:30.825796 10918 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:30.832834 10918 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:30.875823 10919 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:30.877182 10919 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:30.860881 10918 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:30.888065 10918 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:30.889034 10918 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:30.892271 10918 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:30.906213 10918 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:30.913136 10918 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:30.927978 10918 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:30.918931 10919 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:30.945425 10919 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:30.946736 10919 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:30.949872 10919 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.022940 10918 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.024937 10918 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.025990 10919 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:31.027022 10919 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.028946 10918 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.030033 10919 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:31.080010 10919 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:31.081211 10919 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.084553 10919 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.122104 10918 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:31.123907 10918 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.165145 10919 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.172135 10919 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:31.328365 10589 debug-util.cc:398] Leaking SignalData structure 0x7b08002ec060 after lost signal to thread 8673
I20260430 07:58:31.391244 10917 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.392565 10917 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.396154 10917 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.306948 10919 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.508317 10918 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.512589 10923 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:31.517606 10923 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.522344 10921 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:31.524487 10921 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.519591 10922 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:31.519585 10920 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.527568 10920 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.530174 10922 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.536770 10921 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.546270 10925 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.551836 10920 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.557641 10923 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:31.560500 10922 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.547821 10925 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.565213 10917 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:31.566653 10918 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.567540 10918 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.568463 10900 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.567377 10917 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.563418 10923 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.552760 10920 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:31.565488 10919 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.597811 10919 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.598811 10925 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.599489 10925 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.600195 10925 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.592027 10920 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.596050 10923 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.575069 10922 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.601470 10919 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.581595 10900 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.610993 10917 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.612923 10918 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.622555 10922 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.664042 10900 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:31.645453 10918 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.665578 10919 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:31.666085 10918 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.666901 10919 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.675308 10794 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.682905 10022 tablet_copy_service.cc:342] P 48e81934d52a4e93be5171ceea193032: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-805814cfaf2b43ab87c90b2b7b56a211 received from {username='slave'} at 127.0.0.1:53754
I20260430 07:58:31.680125 10917 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:31.683665 10928 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.683964 10022 tablet_copy_service.cc:434] P 48e81934d52a4e93be5171ceea193032: ending tablet copy session d66db415f0f0402fa431349afbf59888-805814cfaf2b43ab87c90b2b7b56a211 on tablet 805814cfaf2b43ab87c90b2b7b56a211 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:31.676609 10794 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.684310 10917 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.696632 10917 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.697654 10919 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.684654 10928 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.676625 10918 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.676676 10920 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.711390 10926 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.712319 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:31.712878 10923 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:31.713471 10566 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:31.715739 10925 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.716475 10925 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:31.722157 10794 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.722728 10922 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.724962 10935 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.726032 10935 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.710551 10920 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:31.729022 10920 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.729471 10935 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.735879 10568 raft_consensus.cc:1217] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:31.736620 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:31.732456 10920 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.735960 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:31.726796 10925 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.746745 10934 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:31.742089 10927 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.742856 10794 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.743952 10918 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.750085 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:31.749845 10794 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.751713 9759 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53852 (request call id 85) took 2147 ms. Trace:
I20260430 07:58:31.747608 10934 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.759480 10928 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.757172 9759 rpcz_store.cc:276] 0430 07:58:29.604384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:29.604585 (+ 201us) service_pool.cc:225] Handling call
0430 07:58:31.751373 (+2146788us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":169,"reported_tablets":1}
I20260430 07:58:31.759480 10794 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:31.761514 10918 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.764070 10918 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.752140 10927 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:31.757639 10929 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.764935 10568 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:31.713649 10926 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.767387 10921 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:31.765863 10929 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.789710 10936 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:31.800961 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:31.801582 10003 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:31.804651 10921 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.805437 10940 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:31.806259 10940 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.809090 10917 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.809959 10917 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.812546 10931 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.797000 10021 tablet_copy_service.cc:342] P 48e81934d52a4e93be5171ceea193032: Request end of tablet copy session d66db415f0f0402fa431349afbf59888-f8946312605d455cbfe6546ad6873ec1 received from {username='slave'} at 127.0.0.1:53754
I20260430 07:58:31.817993 10928 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.859817 10021 tablet_copy_service.cc:434] P 48e81934d52a4e93be5171ceea193032: ending tablet copy session d66db415f0f0402fa431349afbf59888-f8946312605d455cbfe6546ad6873ec1 on tablet f8946312605d455cbfe6546ad6873ec1 with peer d66db415f0f0402fa431349afbf59888
I20260430 07:58:31.854439 10925 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.868474 10917 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.884948 10926 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.813637 10931 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.885505 10936 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:32.034437 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:31.968987 10927 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:32.036595 10939 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.035297 10003 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:32.047511 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:31.886145 10917 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:32.049996 10930 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.888549 10923 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:31.888100 10921 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.911164 10934 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.050866 10004 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:32.026892 10900 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:32.063196 10932 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.064845 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:32.074291 10568 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:32.047286 10937 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:32.046959 10940 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:31.960709 10926 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.016033 10938 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:32.067843 10930 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.070623 10900 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.066306 10923 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:31.950054 10929 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.079173 10932 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.083530 10937 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:31.923214 10928 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:32.095583 10937 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.100164 10900 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:31.922327 10943 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:32.105013 10921 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.109503 10921 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.111143 10932 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.111961 10932 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:32.112468 10921 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.086807 10940 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.113853 10940 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.105079 10917 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.112768 10932 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.118448 10917 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:32.070442 10939 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:32.121548 10930 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:32.124547 10936 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.126977 10923 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.185551 10533 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:32.108435 10929 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.122155 10932 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.223392 9977 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:32.226758 10923 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:32.227878 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 104696us
I20260430 07:58:32.228062 10927 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260430 07:58:32.335624 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 257165us
I20260430 07:58:32.129511 10926 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.340960 10926 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.350194 10955 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:32.350845 10955 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:32.351284 10955 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:32.188512 10936 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.371433 10955 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:32.350206 9828 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.024s sys 0.060s
I20260430 07:58:32.376549 10940 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.342026 10927 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:32.381806 9828 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s user 0.024s sys 0.060s
I20260430 07:58:32.389358 9977 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:32.264957 10938 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:32.384610 10927 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:32.378041 10943 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:32.420351 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:32.420784 10957 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.421144 10565 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:32.421480 10957 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.423626 10957 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.424054 10947 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:32.384967 10929 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:32.431725 10794 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.040s sys 0.055s
W20260430 07:58:32.432458 10794 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s user 0.041s sys 0.055s
I20260430 07:58:32.434989 9977 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:32.399689 10953 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.395164 10533 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:32.407882 10952 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:32.424831 10947 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.391922 10923 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.436977 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:32.534432 10941 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.430621 10939 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.409883 10936 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:32.537607 10928 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:32.548178 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 114871us
I20260430 07:58:32.613512 10960 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:32.613991 10943 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:32.433719 10940 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.612987 10942 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:32.612452 9901 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.039s sys 0.061s
W20260430 07:58:32.612685 9826 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.048s sys 0.056s
W20260430 07:58:32.623059 9901 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.803s user 0.041s sys 0.062s
I20260430 07:58:32.623441 9977 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:32.636749 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.057210 10009 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:32.637849 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.058591 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.059217 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.059929 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.060536 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:33.061848 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 62877.8us
I20260430 07:58:32.636803 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:32.638149 10010 raft_consensus.cc:1240] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:33.066715 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.066793 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.068014 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.060554 10009 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:32.622582 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:32.651255 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:32.634855 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:32.537817 10921 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:32.659875 10923 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.622956 10928 raft_consensus.cc:2804] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.077749 10928 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.078279 10928 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:32.693434 10927 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.084143 10927 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.634155 10929 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.515765 10931 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.662556 10936 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:32.662653 10938 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.610589 10941 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.088595 10980 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.088845 10969 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.632192 10942 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:33.089432 10980 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.622267 10948 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.092440 10942 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.673321 10947 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.655242 10946 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.099740 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:33.100013 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:33.100489 10946 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.101135 10931 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:32.633168 10950 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.102689 10928 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.103652 10950 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.104404 9828 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:33.104418 10567 raft_consensus.cc:1240] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:33.105510 10928 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.105710 10950 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.106232 10950 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.106537 10950 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.106854 10950 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.106990 10978 consensus_queue.cc:1059] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:58:33.107602 9828 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:33.108289 10929 raft_consensus.cc:3055] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:32.618117 10953 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.108333 10934 consensus_queue.cc:1059] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:58:33.110533 10934 raft_consensus.cc:3055] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:33.110839 10934 raft_consensus.cc:740] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:33.111261 10953 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.695883 10955 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:32.609061 10956 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:32.609362 10959 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.113209 10956 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.113883 10959 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.114430 10959 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.115357 10956 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.614256 10960 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.638592 10962 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.118885 10950 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.119030 10962 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.119829 10960 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.121629 10962 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.537505 10952 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:32.843809 9826 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.856s user 0.049s sys 0.056s
I20260430 07:58:33.122886 10952 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:32.898917 10949 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:33.124145 10949 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.125061 10956 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.127028 10947 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.127174 10956 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.127698 10947 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.899039 10964 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.900923 10966 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:32.900095 10965 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.903319 10564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54754 (request call id 120) took 1170 ms. Trace:
I20260430 07:58:32.906236 10968 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:32.906881 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:32.914062 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:32.915982 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:32.918393 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.129244 10956 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.132521 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.132705 10956 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.132916 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.133636 9938 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.133416 10956 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:33.133957 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.134557 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:33.135804 10534 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.129:36679 timed out after 1.572s (SENT) [suppressed 4 similar messages]
I20260430 07:58:33.136233 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:32.918677 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:32.920228 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.137421 10956 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.134994 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.138274 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.139099 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.139719 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.140399 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.141022 10006 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:32.927009 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:32.927430 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:32.927986 9978 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.013s sys 0.001s
W20260430 07:58:32.664458 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 276625us
I20260430 07:58:32.951685 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:32.959460 10589 debug-util.cc:398] Leaking SignalData structure 0x7b08002ec740 after lost signal to thread 9795
I20260430 07:58:32.964547 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:32.975199 10975 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:32.977129 10943 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.129778 10964 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.147153 10943 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.148113 10952 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.129920 10950 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.149605 10943 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.149775 10964 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.977205 10958 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:32.983384 10976 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:32.997996 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:33.002450 9904 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35268 dropped due to backpressure: service queue is full with 50 items
I20260430 07:58:33.006105 10971 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:33.150882 10958 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.151783 10971 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.130093 10965 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:33.130350 10968 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.154260 10971 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.003578 10561 raft_consensus.cc:1217] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:33.008664 10939 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.011372 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.022657 10977 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.022938 10973 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:33.023495 10972 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:33.023800 9827 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.155s user 0.024s sys 0.064s
I20260430 07:58:33.024698 9902 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35268 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 944)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 673)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 926)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 680)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 950)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 932)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 948)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 30)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 682)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 946)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 940)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 31)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 674)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 943)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 941)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 945)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 942)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 675)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 949)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 677)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 678)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 919)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 921)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 922)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 923)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 935)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 936)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 937)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 947)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 951)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 953)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 954)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 683)
I20260430 07:58:33.022679 10974 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.033758 10932 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.035601 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.038802 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.040210 9761 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48098 (request call id 187) took 1324 ms. Trace:
I20260430 07:58:33.146840 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.056384 10944 raft_consensus.cc:2804] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.064661 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.069195 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.074482 10921 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.067972 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.165357 9761 rpcz_store.cc:276] 0430 07:58:31.715847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:31.716222 (+ 375us) service_pool.cc:225] Handling call
0430 07:58:33.040017 (+1323795us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23466,"reported_tablets":1}
I20260430 07:58:33.069705 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.084539 10970 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.089458 10969 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:32.647022 10940 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.191808 10921 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.191465 10950 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.192549 10950 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.092519 10948 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.193357 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.094033 10979 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.096423 10941 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:33.096552 10568 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:33.188555 10943 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.102720 10946 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.108831 10929 raft_consensus.cc:740] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:33.111701 10934 consensus_queue.cc:260] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.109166 10931 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.122113 9759 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53852 (request call id 86) took 1028 ms. Trace:
I20260430 07:58:33.126519 10949 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.199791 10979 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.006590 10957 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:33.144062 9978 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.262s user 0.013s sys 0.006s
I20260430 07:58:33.130959 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.143777 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.141963 9828 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:33.144373 10011 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:33.204309 10957 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.204216 10929 consensus_queue.cc:260] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.206671 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.206970 10929 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:33.131235 9934 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.131832 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.159426 9929 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.208775 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.209419 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.209532 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:33.210129 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.210387 9903 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:33.211167 10949 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.204839 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.211562 10949 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:33.211661 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.212002 10949 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.212249 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:33.165220 9827 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.304s user 0.026s sys 0.065s
I20260430 07:58:33.212917 10918 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.212023 9931 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:33.194672 10969 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.144342 10533 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:33.213536 10918 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.214125 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:33.214404 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 70166.9us
W20260430 07:58:33.055768 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 433770us
W20260430 07:58:33.145650 10589 debug-util.cc:398] Leaking SignalData structure 0x7b0800288300 after lost signal to thread 9887
I20260430 07:58:33.146733 10947 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.142333 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.129446 10923 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.222179 10923 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.168385 10009 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.141126 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.129834 10966 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.202576 10931 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.144659 10959 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.177199 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.179121 9938 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.180748 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.151158 9904 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35268 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 948)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 928)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 30)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 682)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 946)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 940)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 676)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 925)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 31)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 674)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 943)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 941)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 945)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 942)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 675)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 949)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 677)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 678)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 919)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 921)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 922)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 923)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 935)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 936)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 937)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 947)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 951)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 953)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 954)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 683)
I20260430 07:58:33.169518 10958 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:33.167539 10968 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.180609 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.146078 10975 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.169360 10965 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.183115 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.173671 10944 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.130043 10564 rpcz_store.cc:276] 0430 07:58:31.733071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:31.736111 (+ 3040us) service_pool.cc:225] Handling call
0430 07:58:31.736666 (+ 555us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:31.747864 (+ 11198us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:31.747900 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:31.747962 (+ 62us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:31.748009 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:32.902720 (+1154711us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b54005c1a54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:32.902804 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:32.903048 (+ 244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25125760}
I20260430 07:58:33.170218 10973 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.170429 10939 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.179672 10932 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.174834 10977 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.187456 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.180574 10974 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.260205 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:33.167477 10972 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.264825 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.264989 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.188750 10952 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:33.187551 10978 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.180873 10970 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:33.271255 9933 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:33.092954 10980 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.198565 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.199625 10948 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:32.902128 10967 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.194331 10950 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.202309 9759 rpcz_store.cc:276] 0430 07:58:32.093468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:32.094882 (+ 1414us) service_pool.cc:225] Handling call
0430 07:58:33.121974 (+1027092us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":942040,"reported_tablets":2}
W20260430 07:58:33.205847 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 63935.6us
I20260430 07:58:33.141834 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.207587 10957 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.212330 10955 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.150887 10976 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.213554 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.210145 10004 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.220386 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.203539 10940 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.235442 10012 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:33.224289 10928 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.238226 10959 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:33.238322 10568 raft_consensus.cc:1217] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:33.265829 10918 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:33.233721 9977 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:33.239416 10003 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:33.241600 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.205013 10921 consensus_queue.cc:1059] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260430 07:58:33.248775 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.248809 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.285210 9861 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.285328 9860 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.234529 10931 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.260331 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.218452 10943 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.239181 10938 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.246284 10934 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:33.249248 10589 debug-util.cc:398] Leaking SignalData structure 0x7b08002fa860 after lost signal to thread 9962
I20260430 07:58:33.269779 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.266405 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.259699 10944 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.258781 10561 raft_consensus.cc:1217] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260430 07:58:33.236974 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.279219 10923 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.293124 9904 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35278 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 670)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 942)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 675)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 696)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 957)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 690)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 34)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 949)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 37)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 36)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 955)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 35)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 689)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 678)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 919)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 921)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 922)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 923)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 935)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 936)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 937)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 947)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 951)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 953)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 954)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 958)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 960)
I20260430 07:58:33.287168 9863 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.277714 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.299680 10006 raft_consensus.cc:1240] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:33.298007 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.301894 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.302977 9934 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
W20260430 07:58:33.284942 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 51328.7us
I20260430 07:58:33.300846 10979 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.302309 10939 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:33.311542 10589 debug-util.cc:398] Leaking SignalData structure 0x7b0800298860 after lost signal to thread 10036
I20260430 07:58:33.321614 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.311866 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.312003 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.312567 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.313305 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.313925 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:33.314289 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.311621 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:33.313956 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.329752 9860 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.281031 10928 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:33.318101 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.318627 9903 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:33.318939 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.318831 10966 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.286335 9931 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.322171 10974 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.339016 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.341260 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.299232 10938 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.296469 10918 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.326534 9935 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.347599 10918 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:33.348162 10918 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.348369 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.326424 9861 raft_consensus.cc:2468] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:33.327334 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.276484 10972 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:33.330830 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:33.339381 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.332963 9858 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.341585 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.308455 9904 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:35278 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 927)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 942)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 675)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 931)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 671)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 696)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 933)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 924)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 957)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 934)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 690)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 34)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 672)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 949)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 681)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 929)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 930)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 677)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 37)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 959)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 693)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 36)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 688)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 955)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 686)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 35)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 689)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 698)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 697)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 678)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 919)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 920)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 921)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 922)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 923)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 935)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 936)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 937)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 938)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 947)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 951)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 952)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 953)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 954)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 683)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 958)
I20260430 07:58:33.342051 9862 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
W20260430 07:58:33.334110 10589 debug-util.cc:398] Leaking SignalData structure 0x7b080032a560 after lost signal to thread 10595
I20260430 07:58:33.289121 10978 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.326368 10955 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.333747 10928 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.346937 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.329536 10981 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.365691 10006 raft_consensus.cc:2393] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:33.332449 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.371717 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.335958 10979 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.302682 10967 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.331403 10975 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.341609 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.384455 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.395025 9859 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.345063 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:33.388008 10973 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.387861 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.388784 10959 raft_consensus.cc:1064] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:33.389237 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.379951 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:33.392352 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.393181 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.390985 10979 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:33.400213 10977 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.400730 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.389460 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.405299 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.393935 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.409806 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.380538 9932 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.414464 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.415328 9861 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:33.395983 10981 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.417531 10981 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.419504 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.408667 10932 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:33.430241 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.432603 10004 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:33.413486 10979 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.433871 10008 raft_consensus.cc:1240] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:33.425554 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.424834 10938 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.448570 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.452922 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.456313 9858 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:33.457052 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.407622 9930 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.447935 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.461999 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.452554 10566 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:33.441692 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:33.461828 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.462911 9929 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.418278 10955 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:33.467151 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.457768 9859 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:33.409572 10978 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:33.424787 9856 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.450572 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.472856 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.451210 10970 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.482470 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.484911 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:33.484671 10972 consensus_queue.cc:1059] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:58:33.473546 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.437063 10949 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.496729 9860 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:33.488399 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.497568 9859 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:33.487442 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.492043 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.494539 10967 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.495039 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:33.442049 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.495851 10928 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.498116 9932 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.518555 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.519541 9932 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.520238 10918 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.512458 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.522500 10928 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.523556 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:33.523802 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.524695 10949 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.524819 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.525967 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.526536 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.527060 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.528275 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.528191 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.512396 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.529975 9930 raft_consensus.cc:2393] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:33.531024 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.532207 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.532521 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.532904 9936 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.533932 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:33.535071 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.536154 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.537248 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.551817 9938 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:33.552155 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:33.554574 10943 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.520825 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.551301 10918 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:33.558750 10981 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.559669 9930 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:33.567970 10981 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.547873 10944 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.586156 10944 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:33.583827 10979 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:33.587670 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 243415us
I20260430 07:58:33.591002 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.592404 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:33.593782 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:33.594645 10565 raft_consensus.cc:2393] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:33.586879 10931 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.596987 9863 raft_consensus.cc:2393] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:33.602898 10931 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.594455 10979 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.608145 10976 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.611835 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:33.616199 10533 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:33.616592 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.617834 9930 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.618865 10928 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:33.624217 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.627022 9937 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
W20260430 07:58:33.631790 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 323674us
I20260430 07:58:33.633864 9828 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:33.635113 9828 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:33.635967 9828 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:33.636852 10976 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:33.638525 10976 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:33.640815 9828 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:33.642791 9828 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:33.645035 10981 raft_consensus.cc:2749] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:33.632898 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:33.649343 10976 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.649840 10976 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.650367 10976 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.649904 9828 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:33.651827 10981 raft_consensus.cc:2804] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.652582 10981 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.653035 10981 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.652683 9977 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:33.654102 9828 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:33.654307 9977 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:33.655289 10943 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.655833 10943 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.656229 10943 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.658346 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:33.659462 9828 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:33.639017 10931 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.666774 10985 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:33.680347 10931 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.684460 10931 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.661773 10534 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:33.686830 10534 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:33.660202 10944 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:33.688031 10938 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.688441 10938 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.688829 10938 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.689548 10928 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.689999 10928 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.660552 9828 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:33.690296 10981 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.665359 10929 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.693172 10929 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.690392 10928 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.693629 10929 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.693810 10981 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.694540 10981 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.694885 10981 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.695237 10981 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.697124 10534 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:33.697594 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.698005 10927 raft_consensus.cc:2804] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.698346 9861 raft_consensus.cc:2393] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:33.699144 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.700250 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:33.679474 10943 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.683267 9935 raft_consensus.cc:2393] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:33.683442 10918 raft_consensus.cc:1064] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:33.704954 10971 raft_consensus.cc:3055] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:33.706014 10931 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.705070 10928 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.716360 10943 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.666400 10978 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:33.698364 10927 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:33.716650 10971 raft_consensus.cc:740] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:33.724897 10927 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.725610 10971 consensus_queue.cc:260] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.727056 10971 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.728291 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.718910 10918 consensus_queue.cc:237] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.666250 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.708427 10981 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.733440 9860 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:33.734398 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.734617 10981 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.734957 9860 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.735842 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.736825 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.739596 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.720249 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.739692 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:33.740510 9863 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.742792 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.732771 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.691560 9828 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:33.742708 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.718333 10931 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:33.723889 10929 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.723794 10976 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:33.745807 10927 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.741521 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.753315 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.714821 10938 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:33.754108 9860 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:33.748178 9862 raft_consensus.cc:2393] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:33.741596 10534 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:33.749538 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.749752 10981 raft_consensus.cc:2804] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.760453 10981 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:33.760902 10981 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.762748 10927 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:33.764086 10927 raft_consensus.cc:2749] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:33.751251 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:34.804679 9855 raft_consensus.cc:2393] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:33.747460 9863 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
W20260430 07:58:33.747615 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 56129.2us
I20260430 07:58:34.805261 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1197) took 1188 ms. Trace:
I20260430 07:58:34.805871 9828 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:33.755290 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:34.806941 10939 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:33.756779 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:33.756994 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:34.807394 10939 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:34.807420 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1194) took 1191 ms. Trace:
I20260430 07:58:33.745414 10928 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:33.765901 10929 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:34.241740 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 934) took 1312 ms. Trace:
I20260430 07:58:35.073230 9935 rpcz_store.cc:276] 0430 07:58:32.929663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.409552 (+479889us) service_pool.cc:225] Handling call
0430 07:58:34.241403 (+831851us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54001ac054. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:34.241642 (+ 239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":622650624}
I20260430 07:58:33.753509 10976 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:34.280328 9936 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:34.364918 10987 log.cc:927] Time spent T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260430 07:58:34.798516 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1175) took 1218 ms. Trace:
I20260430 07:58:35.075742 9857 rpcz_store.cc:276] 0430 07:58:33.580532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:34.797472 (+1216940us) service_pool.cc:225] Handling call
0430 07:58:34.797894 (+ 422us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:34.798116 (+ 222us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:34.798149 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:34.798192 (+ 43us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:34.798222 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:34.798305 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:34.798458 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.077204 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:34.805536 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41094 (request call id 21) took 1129 ms. Trace:
I20260430 07:58:35.077823 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 678) took 2157 ms. Trace:
I20260430 07:58:34.806970 9828 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.078303 9936 rpcz_store.cc:276] 0430 07:58:32.920373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.552543 (+632170us) service_pool.cc:225] Handling call
0430 07:58:34.264176 (+711633us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.074771 (+810595us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.077128 (+ 2357us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54001ac050. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:35.077720 (+ 592us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1590109568}
W20260430 07:58:35.079126 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 272081us
I20260430 07:58:35.079963 9828 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.080631 10929 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:34.807879 10939 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.081014 10929 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.081578 10929 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:33.760944 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:34.807996 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1208) took 1170 ms. Trace:
I20260430 07:58:35.083730 9856 rpcz_store.cc:276] 0430 07:58:33.637350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.756742 (+119392us) service_pool.cc:225] Handling call
0430 07:58:34.807941 (+1051199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.084903 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.085585 9856 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:35.086098 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1200) took 1468 ms. Trace:
I20260430 07:58:35.086428 9856 rpcz_store.cc:276] 0430 07:58:33.617963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.084586 (+1466623us) service_pool.cc:225] Handling call
0430 07:58:35.086038 (+ 1452us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:34.806146 9855 rpcz_store.cc:276] 0430 07:58:33.616680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.750943 (+134263us) service_pool.cc:225] Handling call
0430 07:58:34.805163 (+1054220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.088060 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.088912 9855 raft_consensus.cc:2393] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:34.808202 9862 rpcz_store.cc:276] 0430 07:58:33.615787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.755046 (+139259us) service_pool.cc:225] Handling call
0430 07:58:34.807334 (+1052288us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:34.815062 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.089375 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1203) took 1459 ms. Trace:
W20260430 07:58:34.859623 9792 debug-util.cc:398] Leaking SignalData structure 0x7b08002a2e20 after lost signal to thread 8673
I20260430 07:58:34.867921 10927 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:35.089581 9932 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:35.089735 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:35.090202 9932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 42) took 1478 ms (client timeout 1610 ms). Trace:
I20260430 07:58:35.090073 10927 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:35.090560 9792 debug-util.cc:398] Leaking SignalData structure 0x7b0800240d00 after lost signal to thread 9795
W20260430 07:58:35.090644 9932 rpcz_store.cc:269] 0430 07:58:33.611788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:34.814697 (+1202909us) service_pool.cc:225] Handling call
0430 07:58:35.090148 (+275451us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.077993 9863 rpcz_store.cc:276] 0430 07:58:33.676292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.742445 (+ 66153us) service_pool.cc:225] Handling call
0430 07:58:34.805483 (+1063038us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:35.091960 9792 debug-util.cc:398] Leaking SignalData structure 0x7b0800064ea0 after lost signal to thread 9887
I20260430 07:58:35.092226 10927 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.092332 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:35.092913 9792 debug-util.cc:398] Leaking SignalData structure 0x7b0800064640 after lost signal to thread 9962
I20260430 07:58:35.092990 9863 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
W20260430 07:58:35.093752 9792 debug-util.cc:398] Leaking SignalData structure 0x7b0800316a40 after lost signal to thread 10036
I20260430 07:58:35.093569 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1207) took 1462 ms. Trace:
W20260430 07:58:35.094439 9792 debug-util.cc:398] Leaking SignalData structure 0x7b080033e900 after lost signal to thread 10595
I20260430 07:58:35.094022 9863 rpcz_store.cc:276] 0430 07:58:33.631512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.092063 (+1460551us) service_pool.cc:225] Handling call
0430 07:58:35.093497 (+ 1434us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.095261 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:34.902791 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:34.914904 10918 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:34.916738 10989 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:34.919741 10944 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:34.932195 10991 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:34.935119 10560 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:34.937839 10992 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:34.939008 10981 consensus_meta.cc:287] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 1.178s user 0.020s sys 0.003s
I20260430 07:58:35.097530 10976 raft_consensus.cc:2804] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.097043 10992 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.097038 10918 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:35.097997 10976 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.098714 10976 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.097082 10981 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:34.941522 10993 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.100775 10993 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.101500 10981 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.102774 10993 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:34.964780 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 1.4033e+06us
I20260430 07:58:35.103243 9862 raft_consensus.cc:2393] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:35.103375 10992 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:35.077865 9857 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:35.106302 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1209) took 1468 ms. Trace:
I20260430 07:58:35.106671 9857 rpcz_store.cc:276] 0430 07:58:33.637823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.076911 (+1439088us) service_pool.cc:225] Handling call
0430 07:58:35.106234 (+ 29323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.107820 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:35.108562 9761 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53852 (request call id 87) took 1663 ms. Trace:
I20260430 07:58:35.110172 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:34.977339 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1174) took 1397 ms. Trace:
I20260430 07:58:34.979795 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.089773 9855 rpcz_store.cc:276] 0430 07:58:33.630333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.087809 (+1457476us) service_pool.cc:225] Handling call
0430 07:58:35.089316 (+ 1507us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:35.122280 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 142505us
I20260430 07:58:35.121583 9859 rpcz_store.cc:276] 0430 07:58:33.580157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:34.976332 (+1396175us) service_pool.cc:225] Handling call
0430 07:58:34.976731 (+ 399us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:34.976929 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:34.976961 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:34.977006 (+ 45us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:34.977036 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:34.977088 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:34.977219 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:34.988065 10938 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:35.123304 10533 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.123931 10938 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.001304 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 952) took 1553 ms. Trace:
I20260430 07:58:35.123956 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:34.985143 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:35.008419 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.015754 10974 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:35.016261 10972 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:35.124537 9854 rpcz_store.cc:276] 0430 07:58:33.448056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.485924 (+ 37868us) service_pool.cc:225] Handling call
0430 07:58:33.507696 (+ 21772us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.000730 (+1493034us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:35.000774 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:35.000841 (+ 67us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:35.000885 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.000988 (+ 103us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:35.001174 (+ 186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.096498 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.126046 10972 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:35.016700 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 106327us
I20260430 07:58:35.017982 10969 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:35.018661 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.124320 10938 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.126912 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.127501 9863 raft_consensus.cc:2393] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:35.020370 10967 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:35.127573 10938 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.127821 9854 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:35.127985 10967 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.128314 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.128818 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.021929 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.026535 10995 raft_consensus.cc:1064] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:35.023972 9978 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.130431 10918 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:35.130867 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 106966us
I20260430 07:58:35.131688 9978 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.131935 10995 consensus_queue.cc:237] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.036347 10959 consensus_queue.cc:237] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.133528 10974 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.135478 10974 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.048517 10997 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.136538 9904 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.136564 10938 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.080962 9828 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:34.964709 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.133188 9978 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.158582 9978 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.159878 9978 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.111546 10944 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.161640 10993 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.162365 10993 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.162701 10993 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.162832 10944 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:35.163118 10993 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.164062 10944 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.164422 10944 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.164786 10944 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.112301 10989 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.165689 10969 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.166833 10989 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:35.087348 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:35.168048 10969 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.168270 9856 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.151453 10972 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.172960 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.113757 10991 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.176051 10991 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:35.176230 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.176774 10005 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.124577 10533 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.180860 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:58:35.180763 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.181494 10006 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:35.185527 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 61021.2us
W20260430 07:58:35.187999 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 107047us
I20260430 07:58:35.188351 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.189802 9828 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:35.190564 10940 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.190932 10940 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.191291 10940 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.108964 9761 rpcz_store.cc:276] 0430 07:58:33.444983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.445127 (+ 144us) service_pool.cc:225] Handling call
0430 07:58:35.108500 (+1663373us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":3}
I20260430 07:58:35.119026 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:35.194000 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.194896 10012 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:35.196637 9828 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:35.198124 9828 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:35.198406 10931 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.198913 10931 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.199328 10931 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.209460 10981 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.210038 10981 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.210407 10981 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.211040 11002 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:35.114156 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1191) took 1500 ms. Trace:
I20260430 07:58:35.213439 9862 rpcz_store.cc:276] 0430 07:58:33.613543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.089468 (+1475925us) service_pool.cc:225] Handling call
0430 07:58:35.113971 (+ 24503us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b54001bb554. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:35.114102 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29165440}
I20260430 07:58:35.214527 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.215113 9862 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:35.216212 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:35.122947 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:35.044991 9934 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 942) took 2113 ms (client timeout 1988 ms). Trace:
I20260430 07:58:35.137060 10997 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.137946 9904 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:35.217978 9934 rpcz_store.cc:269] 0430 07:58:32.932021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.044755 (+2112734us) service_pool.cc:212] Skipping call since client already timed out
0430 07:58:35.044900 (+ 145us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260430 07:58:35.218966 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 81047.2us
I20260430 07:58:35.219583 10997 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.220019 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.220727 9855 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:35.220819 9904 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.138983 10949 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.221446 10949 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.221894 10949 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.222404 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1179) took 1624 ms. Trace:
I20260430 07:58:35.139966 10919 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:35.064039 10534 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.304s user 0.009s sys 0.016s
W20260430 07:58:35.227948 10534 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.468s user 0.010s sys 0.017s
W20260430 07:58:35.228596 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 1.48639e+06us
I20260430 07:58:35.130972 10918 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.141687 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.125034 9859 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:35.234191 9859 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.234819 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1180) took 1636 ms. Trace:
I20260430 07:58:35.235139 9859 rpcz_store.cc:276] 0430 07:58:33.598742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.233730 (+1634988us) service_pool.cc:225] Handling call
0430 07:58:35.234026 (+ 296us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.234577 (+ 551us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.234758 (+ 181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.144079 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.144852 9857 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:35.144776 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.238363 10009 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:35.146210 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:35.146783 9858 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1173) took 1586 ms (client timeout 1774 ms). Trace:
W20260430 07:58:35.239882 9858 rpcz_store.cc:269] 0430 07:58:33.560056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.008123 (+1448067us) service_pool.cc:225] Handling call
0430 07:58:35.146722 (+138599us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.145632 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.241906 9858 raft_consensus.cc:1240] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.142755 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.150719 10976 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:35.153812 10967 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.153765 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.155314 10939 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.155714 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:35.149485 10928 raft_consensus.cc:2804] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election won for term 2
W20260430 07:58:35.120805 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:35.217898 11002 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.194152 10560 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:35.189903 10927 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.241036 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1182) took 1641 ms. Trace:
I20260430 07:58:35.208984 9931 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
W20260430 07:58:35.261950 9931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 40) took 1700 ms (client timeout 1596 ms). Trace:
I20260430 07:58:35.261565 11002 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.262224 9860 raft_consensus.cc:2393] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:35.262478 10976 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.243019 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1183) took 1643 ms. Trace:
I20260430 07:58:35.264820 9858 rpcz_store.cc:276] 0430 07:58:33.599557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.241011 (+1641454us) service_pool.cc:225] Handling call
0430 07:58:35.241361 (+ 350us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.242565 (+ 1204us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.242969 (+ 404us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.254709 9978 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.266659 9978 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.267133 10010 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:35.089124 10929 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:35.263605 9860 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1205) took 1632 ms (client timeout 2016 ms). Trace:
W20260430 07:58:35.268234 9860 rpcz_store.cc:269] 0430 07:58:33.630934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.756559 (+125625us) service_pool.cc:225] Handling call
0430 07:58:35.263445 (+1506886us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b540010c054. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:35.263558 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1044479360}
I20260430 07:58:35.269052 9861 raft_consensus.cc:2393] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:35.269642 11002 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:35.270082 9861 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1192) took 1656 ms (client timeout 2073 ms). Trace:
I20260430 07:58:35.252942 10974 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.270907 10974 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.271250 10939 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:35.270792 9861 rpcz_store.cc:269] 0430 07:58:33.613871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.760693 (+146822us) service_pool.cc:225] Handling call
0430 07:58:35.269904 (+1509211us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b5400397454. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:35.270026 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":425054464}
I20260430 07:58:35.271451 10974 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.260947 9857 rpcz_store.cc:276] 0430 07:58:33.599292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.238745 (+1639453us) service_pool.cc:225] Handling call
0430 07:58:35.239859 (+ 1114us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.240303 (+ 444us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:35.240444 (+ 141us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:35.240545 (+ 101us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:58:35.240599 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.240709 (+ 110us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:35.240932 (+ 223us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:35.273641 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 75547us
I20260430 07:58:35.274292 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1186) took 1674 ms. Trace:
W20260430 07:58:35.273685 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 75769.1us
I20260430 07:58:35.274968 10928 raft_consensus.cc:697] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:35.277629 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.277742 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1189) took 1676 ms. Trace:
I20260430 07:58:35.274662 9860 rpcz_store.cc:276] 0430 07:58:33.600356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.269049 (+1668693us) service_pool.cc:225] Handling call
0430 07:58:35.273170 (+ 4121us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.273970 (+ 800us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:35.273996 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:35.274047 (+ 51us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:35.274080 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.274141 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:35.274242 (+ 101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.279187 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.258949 10534 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.286995 10006 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
W20260430 07:58:35.280922 9959 debug-util.cc:398] Leaking SignalData structure 0x7b0800164cc0 after lost signal to thread 8673
I20260430 07:58:35.284809 9903 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:35.281180 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.281864 9854 raft_consensus.cc:1240] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.306648 10010 raft_consensus.cc:2393] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:35.306959 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1187) took 1706 ms. Trace:
W20260430 07:58:35.307435 9959 debug-util.cc:398] Leaking SignalData structure 0x7b08001683a0 after lost signal to thread 9795
I20260430 07:58:35.307334 9854 rpcz_store.cc:276] 0430 07:58:33.600612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.269904 (+1669292us) service_pool.cc:225] Handling call
0430 07:58:35.281462 (+ 11558us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:35.281676 (+ 214us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.306665 (+ 24989us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.306897 (+ 232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7599872}
W20260430 07:58:35.308560 9959 debug-util.cc:398] Leaking SignalData structure 0x7b0800168460 after lost signal to thread 9809
I20260430 07:58:35.309953 10949 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.285939 9857 rpcz_store.cc:276] 0430 07:58:33.601184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.273044 (+1671860us) service_pool.cc:225] Handling call
0430 07:58:35.275954 (+ 2910us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.277380 (+ 1426us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:35.277414 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:35.277462 (+ 48us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:35.277492 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.277562 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:35.277689 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.311674 9854 raft_consensus.cc:1240] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.312167 9857 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.312978 9854 raft_consensus.cc:1240] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.283180 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1188) took 1682 ms. Trace:
I20260430 07:58:35.313949 9861 rpcz_store.cc:276] 0430 07:58:33.600867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.272341 (+1671474us) service_pool.cc:225] Handling call
0430 07:58:35.282430 (+ 10089us) spinlock_profiling.cc:243] Waited 990 us on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:35.282684 (+ 254us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.282874 (+ 190us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:35.282899 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:35.282938 (+ 39us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:58:35.282964 (+ 26us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.283016 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:35.283129 (+ 113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2278144}
I20260430 07:58:35.315088 10949 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.316075 9861 raft_consensus.cc:1240] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.283169 9856 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:35.283895 10934 consensus_queue.cc:1059] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:58:35.317611 10967 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.318418 10934 raft_consensus.cc:3055] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:35.318893 10934 raft_consensus.cc:740] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:35.302498 10955 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.319759 10934 consensus_queue.cc:260] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.280258 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1184) took 1680 ms. Trace:
I20260430 07:58:35.321435 9863 rpcz_store.cc:276] 0430 07:58:33.599807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.264274 (+1664467us) service_pool.cc:225] Handling call
0430 07:58:35.279339 (+ 15065us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:35.279550 (+ 211us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.279740 (+ 190us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:35.279764 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:35.280008 (+ 244us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:35.280043 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.280104 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:35.280205 (+ 101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2717056}
I20260430 07:58:35.302757 10972 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:35.286317 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 65548.9us
I20260430 07:58:35.289753 9828 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:35.323501 9863 raft_consensus.cc:1240] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.323930 9904 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.324846 9828 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.325245 9904 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.326282 9904 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.286082 9934 raft_consensus.cc:2376] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2. Re-sending same reply.
I20260430 07:58:35.295704 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1190) took 1694 ms. Trace:
I20260430 07:58:35.296566 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1181) took 1697 ms. Trace:
I20260430 07:58:35.327745 9828 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.327839 9859 rpcz_store.cc:276] 0430 07:58:33.599021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.237164 (+1638143us) service_pool.cc:225] Handling call
0430 07:58:35.263875 (+ 26711us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.286602 (+ 22727us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:35.286630 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:35.286671 (+ 41us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:35.296336 (+ 9665us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.296401 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:35.296493 (+ 92us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:35.296854 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 67566.4us
I20260430 07:58:35.286314 10928 consensus_queue.cc:237] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.297696 10944 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.338999 10944 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:35.298061 10927 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.339893 10927 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.340504 10927 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.299321 10005 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:35.284741 10940 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.270174 10929 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.335906 9863 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:35.341349 10977 consensus_queue.cc:1059] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:35.353778 10939 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.305706 9978 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.222826 9855 rpcz_store.cc:276] 0430 07:58:33.598453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.221488 (+1623035us) service_pool.cc:225] Handling call
0430 07:58:35.221830 (+ 342us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.222042 (+ 212us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:35.222070 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:35.222116 (+ 46us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:35.222155 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.222223 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:35.222347 (+ 124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.336339 10534 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.358088 10534 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.359208 10997 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.359588 10997 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.360114 10997 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.362922 10940 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.342468 10967 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.364482 10967 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.364897 10967 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.330348 9862 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:35.327539 9860 rpcz_store.cc:276] 0430 07:58:33.601532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.286640 (+1685108us) service_pool.cc:225] Handling call
0430 07:58:35.286943 (+ 303us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.295311 (+ 8368us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:35.295340 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:35.295393 (+ 53us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:58:35.295463 (+ 70us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.295527 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:35.295647 (+ 120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.331933 9859 raft_consensus.cc:1240] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.327860 9904 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.332317 9828 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:35.367269 9857 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.368290 9828 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:35.262406 9931 rpcz_store.cc:269] 0430 07:58:33.561858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:33.565443 (+ 3585us) service_pool.cc:225] Handling call
0430 07:58:35.261849 (+1696406us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.370136 9828 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.317687 9861 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.333072 9858 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.334241 10921 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.334342 11002 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:35.335445 9959 debug-util.cc:398] Leaking SignalData structure 0x7b080035f260 after lost signal to thread 9887
W20260430 07:58:35.376835 9959 debug-util.cc:398] Leaking SignalData structure 0x7b080035f280 after lost signal to thread 9888
I20260430 07:58:35.331416 10969 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.379508 10996 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:35.382485 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 76915.3us
I20260430 07:58:35.383183 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1185) took 1783 ms. Trace:
I20260430 07:58:35.383533 9978 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.383682 9858 rpcz_store.cc:276] 0430 07:58:33.600061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.265605 (+1665544us) service_pool.cc:225] Handling call
0430 07:58:35.332909 (+ 67304us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:35.371087 (+ 38178us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:35.382834 (+ 11747us) spinlock_profiling.cc:243] Waited 66.9 ms on lock 0x7b540010cf50. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:35.383091 (+ 257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153964288}
I20260430 07:58:35.384790 9978 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.332423 10995 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.393795 11003 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:35.394121 10534 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.395064 11003 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.395457 11003 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.395931 11003 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.390775 10940 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.373251 10976 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:35.375334 9856 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41094 (request call id 24) took 1630 ms (client timeout 2032 ms). Trace:
W20260430 07:58:35.429577 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 318378us
I20260430 07:58:35.455978 9978 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.392529 10995 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.428988 10955 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.459583 9978 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:35.467191 10999 log.cc:927] Time spent T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260430 07:58:35.474845 10989 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.464565 10995 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:35.432780 9856 rpcz_store.cc:269] 0430 07:58:33.740880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.087084 (+1346204us) service_pool.cc:225] Handling call
0430 07:58:35.371149 (+284065us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:35.398983 10994 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.429199 10934 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.398056 11002 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.390760 10918 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:35.503016 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 43606us
I20260430 07:58:35.504042 9978 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:35.501044 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 113547us
W20260430 07:58:35.495344 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 167460us
I20260430 07:58:35.434701 10977 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.514948 10997 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.458851 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:35.518874 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 85759.8us
I20260430 07:58:35.518477 10928 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.520459 10533 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:35.434128 9959 debug-util.cc:398] Leaking SignalData structure 0x7b080034daa0 after lost signal to thread 9962
I20260430 07:58:35.502727 11002 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.508512 9904 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.434092 10927 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.499588 10534 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.565477 10977 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.566630 11002 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.526885 10949 raft_consensus.cc:3055] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:35.569711 10949 raft_consensus.cc:740] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:35.570942 10949 consensus_queue.cc:260] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.530097 10997 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.573779 10997 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.574331 10997 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.574795 10997 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.530602 10938 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.531965 10928 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.541499 9828 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.546337 9978 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.547721 10918 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.512002 10919 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.535750 10981 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.579267 10919 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.522154 10931 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.579846 10919 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:35.551568 9959 debug-util.cc:398] Leaking SignalData structure 0x7b08003052e0 after lost signal to thread 10036
I20260430 07:58:35.581894 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.584807 9978 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:35.585538 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:35.585366 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 76948.7us
I20260430 07:58:35.586337 9828 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:35.590277 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 215262us
I20260430 07:58:35.590317 9978 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.591320 11007 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:35.573215 10949 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.593745 10931 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.589257 10939 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.586437 10976 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.533859 10993 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.569074 10927 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:35.585718 10998 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.583232 10981 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:35.600340 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 80005.7us
I20260430 07:58:35.597538 9904 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.602093 10952 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.602432 10972 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.539408 10967 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.591598 10928 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:35.607252 9978 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.607995 9828 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.636859 10928 raft_consensus.cc:697] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:35.628423 9761 catalog_manager.cc:5671] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 2 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:35.637603 10976 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.637980 10928 consensus_queue.cc:237] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.639422 9828 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.596869 10955 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.624768 9904 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:35.609542 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 107731us
I20260430 07:58:35.641260 9828 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.641718 9978 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.640261 10955 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.640367 10976 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:35.642915 10955 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.643154 10976 raft_consensus.cc:697] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:35.644124 10976 consensus_queue.cc:237] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:35.586692 9959 debug-util.cc:398] Leaking SignalData structure 0x7b080028b1c0 after lost signal to thread 10595
I20260430 07:58:35.645535 10952 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260430 07:58:35.645792 9959 debug-util.cc:398] Leaking SignalData structure 0x7b0800361940 after lost signal to thread 10957
I20260430 07:58:35.645949 10952 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.646358 10952 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.649056 9978 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:35.649827 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 53204.6us
W20260430 07:58:35.646368 9959 debug-util.cc:398] Leaking SignalData structure 0x7b080035db80 after lost signal to thread 10982
I20260430 07:58:35.626971 10977 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.650447 10981 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.680029 10565 raft_consensus.cc:1275] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:35.680869 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.682363 9931 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.683773 9903 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:35.676622 10560 raft_consensus.cc:1240] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.685712 11004 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.686103 11004 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:35.686436 11004 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.687954 9903 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:35.672987 9978 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.675110 10918 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.676594 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.677752 10959 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.678758 10993 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:35.679333 10967 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:35.689244 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.697177 10977 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.700273 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.697471 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.697078 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.698228 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.704021 10009 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:35.692566 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 67856.9us
I20260430 07:58:35.689568 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:35.616690 10995 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.691783 9903 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:35.697705 10989 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:35.711014 10989 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.711861 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.712968 10989 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:35.718029 9903 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:35.722427 9904 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.692806 10959 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.691157 10998 raft_consensus.cc:2749] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.696933 10949 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.690289 10918 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.730181 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.730530 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.730800 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.770539 9938 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.733139 10993 consensus_queue.cc:1059] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:35.719729 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.737242 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.738775 10944 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:35.739918 10959 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.727242 10972 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.725623 11008 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election won for term 2
W20260430 07:58:35.729077 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 267741us
I20260430 07:58:35.742421 10009 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:35.745213 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.746233 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.752166 9904 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.785764 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:35.786047 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:35.786381 9904 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.749378 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.773187 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.777595 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.787870 9904 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.745339 10969 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:35.744876 10994 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.746731 10952 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.789882 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.788103 9933 raft_consensus.cc:2393] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:35.791738 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:35.777612 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 55216us
I20260430 07:58:35.788938 9828 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:35.763652 10949 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:35.773924 10918 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.791775 9858 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:35.812837 9977 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:35.793954 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:35.794363 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:35.794494 10565 raft_consensus.cc:1240] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.794626 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:35.795516 10566 raft_consensus.cc:1240] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.797595 10994 consensus_queue.cc:1059] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:35.813522 11007 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:35.820358 10997 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.821203 10991 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.797924 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.798686 10010 raft_consensus.cc:2393] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:35.798482 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.799103 10952 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:35.801420 10979 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.829262 10994 raft_consensus.cc:3055] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:35.827227 11007 raft_consensus.cc:697] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:35.821861 10991 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:35.829411 10979 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.801580 9937 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:35.791416 10967 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.810396 10995 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:35.802685 10007 raft_consensus.cc:1240] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.797621 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.803378 9930 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:35.791270 10992 consensus_queue.cc:1059] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260430 07:58:35.804322 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.791465 10968 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260430 07:58:35.780220 10974 consensus_meta.cc:287] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.508s user 0.010s sys 0.003s
I20260430 07:58:35.804906 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.805361 10534 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.805507 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.790884 9904 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.805899 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.806366 11008 raft_consensus.cc:697] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:35.807650 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:35.810474 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:35.851238 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 58076.3us
I20260430 07:58:35.793736 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:35.816459 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.802492 10969 raft_consensus.cc:697] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:35.833129 10991 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.815263 10949 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.833935 10979 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.839339 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.829818 10994 raft_consensus.cc:740] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:35.840193 10566 raft_consensus.cc:1275] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:35.840950 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.829754 10997 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:35.844136 10998 raft_consensus.cc:3055] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:35.869295 10998 raft_consensus.cc:740] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:35.869004 10969 consensus_queue.cc:237] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.869623 10994 consensus_queue.cc:260] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.851540 9978 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:58:35.852064 10534 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.827658 10993 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260430 07:58:35.871634 10534 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.825012 9854 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.854499 9933 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:35.778357 10955 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.859974 10560 raft_consensus.cc:2393] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:35.856966 10928 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:35.855669 9759 catalog_manager.cc:5671] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 2 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:35.868875 9929 raft_consensus.cc:2393] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
W20260430 07:58:35.852696 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 61845.3us
I20260430 07:58:35.856688 10972 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:35.850726 10968 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.884348 10534 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.857025 9931 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:35.895153 10534 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:35.891064 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.892684 9904 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.847853 11002 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.897235 9904 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.900441 9935 raft_consensus.cc:1240] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.902698 9933 raft_consensus.cc:1240] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.870182 10998 consensus_queue.cc:260] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.870683 11011 consensus_queue.cc:1059] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260430 07:58:35.854441 10974 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.893560 10976 consensus_queue.cc:1059] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260430 07:58:35.914094 9933 raft_consensus.cc:1275] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:35.861766 11008 consensus_queue.cc:237] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.893834 10927 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.920989 9904 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:35.882543 11009 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.897780 10992 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:35.903832 10934 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.925045 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:35.832538 11007 consensus_queue.cc:237] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:35.904700 10938 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:35.929013 9937 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:35.899874 10972 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:35.899663 10968 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:35.926481 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 102917us
I20260430 07:58:35.911525 10997 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.915934 10994 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.910640 10928 raft_consensus.cc:697] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
W20260430 07:58:35.935645 9977 leader_election.cc:336] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.767s (SENT)
I20260430 07:58:35.949631 10921 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.936607 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:35.950222 11013 log.cc:927] Time spent T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260430 07:58:36.007288 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:36.008019 9931 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:36.008368 9933 raft_consensus.cc:1240] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.008893 10977 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:35.950139 10998 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.009680 9936 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.926017 11006 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:35.910070 10944 consensus_queue.cc:1059] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260430 07:58:35.914448 10989 consensus_queue.cc:1059] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260430 07:58:35.935241 10007 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.001734 9935 raft_consensus.cc:1275] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:36.004608 10955 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:36.009550 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:35.898965 11003 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:35.936997 10972 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.076086 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:36.000790 9977 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:36.077718 10004 raft_consensus.cc:1240] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:35.928248 10971 consensus_queue.cc:1059] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260430 07:58:36.023072 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:36.024587 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:36.008980 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:36.075050 11010 log.cc:927] Time spent T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.229s user 0.000s sys 0.010s
I20260430 07:58:36.043165 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:36.095432 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 159724us
I20260430 07:58:36.071095 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:36.074509 10010 raft_consensus.cc:1240] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.001967 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:35.968770 11011 consensus_queue.cc:1059] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
W20260430 07:58:36.081881 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 186738us
I20260430 07:58:36.086390 9938 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
W20260430 07:58:36.047838 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 137201us
I20260430 07:58:36.059777 10012 raft_consensus.cc:1240] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.071251 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:36.094830 10566 raft_consensus.cc:1240] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.105902 10981 consensus_queue.cc:1059] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:36.046972 10919 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.106436 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:36.105371 9977 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:36.114606 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:36.115986 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:36.116230 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.117794 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:36.119129 10995 raft_consensus.cc:3055] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.104239 10944 raft_consensus.cc:3055] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.150005 9862 raft_consensus.cc:1240] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.150992 10568 raft_consensus.cc:1240] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.152498 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:36.157567 9861 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.102216 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:36.119030 10561 raft_consensus.cc:1240] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.185505 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.117022 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:36.186822 9935 raft_consensus.cc:1240] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.186986 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.134159 10952 raft_consensus.cc:3055] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.115916 10560 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:36.155982 10971 consensus_queue.cc:1059] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260430 07:58:36.146624 11002 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:36.105528 10968 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.160000 10931 raft_consensus.cc:3055] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.234136 10919 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:36.210114 10944 raft_consensus.cc:740] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:36.155371 10955 raft_consensus.cc:2749] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:36.183518 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:36.256644 10971 consensus_queue.cc:1059] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260430 07:58:36.236491 10952 raft_consensus.cc:740] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:36.309851 9933 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.192022 10995 raft_consensus.cc:740] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:36.240425 10974 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:36.132524 10928 consensus_queue.cc:237] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:36.277649 10931 raft_consensus.cc:740] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:36.314050 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:36.311169 10921 raft_consensus.cc:3055] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.321735 10955 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:36.344376 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.346932 9859 raft_consensus.cc:1240] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.319967 10992 raft_consensus.cc:697] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:36.351421 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.353979 9858 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.358264 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.365933 10974 raft_consensus.cc:3055] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.366487 10968 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:36.366659 10974 raft_consensus.cc:740] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:36.367576 10974 consensus_queue.cc:260] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.368899 10931 consensus_queue.cc:260] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.369576 9857 raft_consensus.cc:1240] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.370165 10931 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.335929 10921 raft_consensus.cc:740] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:36.384557 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:36.354072 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:36.392205 10567 raft_consensus.cc:2393] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
W20260430 07:58:36.392833 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 56725.2us
I20260430 07:58:36.355160 10955 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:36.386538 10939 raft_consensus.cc:3055] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.388898 11017 raft_consensus.cc:3055] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.396899 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:36.352155 10952 consensus_queue.cc:260] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.377394 11003 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:36.373214 10995 consensus_queue.cc:260] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.400748 10921 consensus_queue.cc:260] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.410420 9933 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:36.402678 10939 raft_consensus.cc:740] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:58:36.396672 10918 consensus_meta.cc:287] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.570s user 0.012s sys 0.015s
I20260430 07:58:36.397365 10919 raft_consensus.cc:3055] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.378381 10944 consensus_queue.cc:260] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.419310 10012 raft_consensus.cc:2393] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:36.408762 10566 raft_consensus.cc:1240] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.409528 11017 raft_consensus.cc:740] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:36.420827 10995 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.420838 10952 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.398671 11009 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:36.421494 11023 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:36.419910 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:36.436077 9904 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.002s sys 0.023s
I20260430 07:58:36.423987 9855 raft_consensus.cc:1240] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:36.465077 9904 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s user 0.004s sys 0.023s
W20260430 07:58:36.450129 10979 consensus_meta.cc:287] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.581s user 0.011s sys 0.005s
I20260430 07:58:36.432840 10921 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.422087 10919 raft_consensus.cc:740] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:36.454510 10949 consensus_meta.cc:287] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.574s user 0.018s sys 0.000s
W20260430 07:58:36.466015 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 544847us
W20260430 07:58:36.433349 10967 consensus_meta.cc:287] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.594s user 0.008s sys 0.008s
I20260430 07:58:36.433177 10939 consensus_queue.cc:260] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:36.462208 10934 consensus_meta.cc:287] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.529s user 0.009s sys 0.008s
I20260430 07:58:36.452751 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:36.424255 10918 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.489370 9904 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:36.491020 9904 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:36.492826 10934 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:36.480263 11020 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260430 07:58:36.493688 10934 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.469071 11009 raft_consensus.cc:697] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:36.476717 9931 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:36.486398 10567 raft_consensus.cc:2393] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:36.480358 10967 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:36.472311 10979 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.511893 10979 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:36.512125 10998 consensus_meta.cc:287] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.502s user 0.010s sys 0.000s
I20260430 07:58:36.513651 10008 raft_consensus.cc:1240] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.498874 10977 consensus_queue.cc:1059] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260430 07:58:36.494678 10968 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.516780 10918 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:36.506959 10561 raft_consensus.cc:2393] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:36.480087 10949 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.523461 9937 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.466498 10992 consensus_queue.cc:237] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:36.505102 10939 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.509025 9930 raft_consensus.cc:1240] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:36.529570 10959 consensus_meta.cc:287] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.733s user 0.010s sys 0.004s
I20260430 07:58:36.530063 10959 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.530516 11020 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:58:36.533006 10003 raft_consensus.cc:1240] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.528609 10967 raft_consensus.cc:3055] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.552196 10955 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:36.555624 10967 raft_consensus.cc:740] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:36.573714 10967 consensus_queue.cc:260] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.574764 10967 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.507867 11017 consensus_queue.cc:260] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.578992 11017 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.581574 10944 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.582084 10959 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:36.569747 11009 consensus_queue.cc:237] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:36.589648 11004 consensus_meta.cc:287] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.903s user 0.015s sys 0.009s
I20260430 07:58:36.590167 11004 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.545352 10919 consensus_queue.cc:260] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:36.591260 10938 consensus_meta.cc:287] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.657s user 0.009s sys 0.000s
I20260430 07:58:36.593627 9932 raft_consensus.cc:2393] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:58:36.608217 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 502919us
I20260430 07:58:36.609122 9977 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:36.610841 10534 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:36.628786 10959 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:36.632061 10959 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:36.632530 10959 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.632570 10919 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.633754 10952 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:36.633785 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.609977 10938 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:58:36.628175 10994 consensus_meta.cc:287] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.627s user 0.012s sys 0.020s
W20260430 07:58:36.651134 9901 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.019s sys 0.038s
W20260430 07:58:36.647773 10972 consensus_meta.cc:287] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.572s user 0.012s sys 0.000s
W20260430 07:58:36.658294 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 504715us
I20260430 07:58:36.640091 10949 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:36.655572 9901 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.640s user 0.019s sys 0.039s
I20260430 07:58:36.641913 11020 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:36.659377 9903 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.009s sys 0.022s
W20260430 07:58:36.668311 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:36.668604 9903 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s user 0.010s sys 0.022s
W20260430 07:58:36.669091 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 564805us
I20260430 07:58:36.671133 9977 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:36.675563 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:36.676544 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:36.677975 10994 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:36.678939 10994 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:36.679363 10994 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.680555 10974 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.684304 10959 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:36.654832 10003 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:58:36.647979 10955 consensus_queue.cc:1059] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260430 07:58:36.686734 10955 raft_consensus.cc:3055] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.686893 10959 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:36.687168 10955 raft_consensus.cc:740] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:36.652094 11018 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:36.694926 9977 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:36.697499 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:36.710242 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:36.688961 9854 consensus_meta.cc:287] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.779s user 0.015s sys 0.009s
W20260430 07:58:36.721135 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:36.711135 10007 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.724591 10952 consensus_queue.cc:1048] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:36.725455 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:36.727013 10005 raft_consensus.cc:1275] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:36.664798 10972 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:36.731814 9978 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.010s sys 0.027s
W20260430 07:58:36.732924 9978 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.545s user 0.010s sys 0.027s
I20260430 07:58:36.732955 10005 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:36.722278 10959 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260430 07:58:36.734602 10565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s user 0.012s sys 0.044s
I20260430 07:58:36.738058 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:36.711036 11004 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:36.743743 10005 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:36.727813 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:36.746755 10008 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:36.718477 9854 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:36.697679 10918 consensus_queue.cc:1059] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260430 07:58:36.752053 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:36.754096 9934 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.766850 10004 raft_consensus.cc:1275] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:36.722957 11007 consensus_queue.cc:1059] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260430 07:58:36.687829 11020 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:36.744452 10959 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:58:36.752681 10989 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.000s sys 0.027s
I20260430 07:58:36.699905 10955 consensus_queue.cc:260] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.756963 10012 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.780414 10534 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:36.733952 10998 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
W20260430 07:58:36.778223 10991 consensus_meta.cc:287] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.907s user 0.022s sys 0.014s
I20260430 07:58:36.802204 9978 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:36.787819 10989 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.690s user 0.000s sys 0.028s
I20260430 07:58:36.777251 10940 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:36.820447 10940 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:36.822605 10940 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:36.822503 10011 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:36.781574 10959 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:36.781788 10969 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.009s sys 0.017s
I20260430 07:58:36.809616 11031 raft_consensus.cc:3055] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.800990 11036 consensus_queue.cc:1059] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260430 07:58:36.814098 9978 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:36.806893 10563 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.770092 10949 consensus_queue.cc:1059] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260430 07:58:36.817041 10938 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:58:36.756496 10952 consensus_queue.cc:1059] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
W20260430 07:58:36.837975 10969 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s user 0.009s sys 0.018s
I20260430 07:58:36.828459 10003 raft_consensus.cc:606] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:58:36.859134 10003 raft_consensus.cc:614] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:58:36.797989 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:36.806936 11018 raft_consensus.cc:2804] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:36.858733 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:36.841064 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 337696us
W20260430 07:58:36.841681 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 93108.8us
I20260430 07:58:36.797405 10971 consensus_queue.cc:1059] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260430 07:58:36.825379 11024 consensus_queue.cc:1059] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
W20260430 07:58:36.833073 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:36.839746 10955 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:36.838217 11031 raft_consensus.cc:740] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:36.857156 10939 raft_consensus.cc:1064] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:36.859175 10533 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:36.870297 10952 raft_consensus.cc:3055] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.816174 11011 consensus_queue.cc:1059] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260430 07:58:36.870411 9854 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:36.811012 10991 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:36.861822 11023 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260430 07:58:36.886420 9827 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.014s sys 0.037s
W20260430 07:58:36.888185 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 132865us
I20260430 07:58:36.815289 10994 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.874749 11018 raft_consensus.cc:697] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
W20260430 07:58:36.897663 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:36.895735 9827 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.581s user 0.014s sys 0.037s
I20260430 07:58:36.873811 11002 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260430 07:58:36.905838 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:36.874405 10918 consensus_queue.cc:1059] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260430 07:58:36.875995 10981 consensus_queue.cc:1048] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260430 07:58:36.886668 10952 raft_consensus.cc:740] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:36.925396 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:36.933173 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:36.899812 10928 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:36.896683 10931 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:36.893326 10972 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:36.906399 11019 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:36.945216 10944 consensus_queue.cc:1059] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260430 07:58:36.931690 11021 consensus_queue.cc:1059] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260430 07:58:36.898741 11036 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260430 07:58:36.956410 10969 raft_consensus.cc:3055] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.910476 11033 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:36.957991 9859 raft_consensus.cc:1240] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:36.942714 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:36.958822 10565 raft_consensus.cc:2393] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:36.894073 10940 consensus_queue.cc:1059] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260430 07:58:36.958026 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:36.954041 11006 consensus_meta.cc:287] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.907s user 0.013s sys 0.010s
I20260430 07:58:36.966015 9860 raft_consensus.cc:2393] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
W20260430 07:58:36.941644 11034 log.cc:927] Time spent T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced: Append to log took a long time: real 0.084s user 0.000s sys 0.002s
I20260430 07:58:36.957039 10977 raft_consensus.cc:3055] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.956847 10969 raft_consensus.cc:740] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:58:36.922111 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 73246.3us
I20260430 07:58:36.928200 11007 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:36.972196 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.974121 10989 raft_consensus.cc:3055] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.974215 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.974794 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:37.019515 9863 raft_consensus.cc:2393] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:36.974876 11033 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.975414 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.976130 10949 raft_consensus.cc:3055] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:36.978705 10940 consensus_queue.cc:1059] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:36.979593 11006 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:36.980248 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.927374 10998 consensus_queue.cc:1059] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260430 07:58:37.016609 9903 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:37.017817 9861 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.019556 9862 raft_consensus.cc:2393] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:37.024827 10952 consensus_queue.cc:260] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:36.922034 11008 consensus_queue.cc:1059] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260430 07:58:36.951716 10931 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:36.971328 10977 raft_consensus.cc:740] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:36.948865 11011 consensus_queue.cc:1048] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
W20260430 07:58:36.998076 10934 consensus_meta.cc:287] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.504s user 0.008s sys 0.005s
I20260430 07:58:36.965585 11020 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:36.971911 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:36.925004 11031 consensus_queue.cc:260] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.018422 10989 raft_consensus.cc:740] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:37.070183 9760 catalog_manager.cc:5671] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 3, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:36.971240 10928 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260430 07:58:37.099964 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:37.094987 11002 raft_consensus.cc:3055] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
W20260430 07:58:37.070389 10921 consensus_meta.cc:287] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.597s user 0.009s sys 0.003s
I20260430 07:58:37.110090 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:36.987922 10972 consensus_queue.cc:1059] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
I20260430 07:58:37.123212 11031 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.125610 10931 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:37.098940 10934 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:37.126067 11017 consensus_meta.cc:287] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.547s user 0.009s sys 0.003s
I20260430 07:58:37.126659 10563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54770 (request call id 238) took 1015 ms. Trace:
I20260430 07:58:37.126992 10994 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:37.128240 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:37.051084 10938 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20260430 07:58:37.129729 10991 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:37.144963 11038 log.cc:927] Time spent T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.086s user 0.001s sys 0.001s
W20260430 07:58:37.082700 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.033553 10949 raft_consensus.cc:740] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:37.131359 10003 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2947a88351384118b009f71f871ab9c8"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "d66db415f0f0402fa431349afbf59888"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:58:37.045424 11022 consensus_queue.cc:1059] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.146s
I20260430 07:58:37.051610 11018 consensus_queue.cc:237] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.171983 10972 raft_consensus.cc:3055] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.172101 11022 raft_consensus.cc:3055] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.171917 10949 consensus_queue.cc:260] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.172513 10972 raft_consensus.cc:740] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:37.172618 11022 raft_consensus.cc:740] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:37.173549 10949 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.173455 10972 consensus_queue.cc:260] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.173619 11022 consensus_queue.cc:260] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.152458 10921 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:37.070854 10977 consensus_queue.cc:260] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.174965 10921 raft_consensus.cc:697] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:37.175612 10977 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.175942 10921 consensus_queue.cc:237] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.053226 10981 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:37.177553 10981 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.179919 10981 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:37.134784 11027 raft_consensus.cc:3055] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.050120 10939 consensus_queue.cc:237] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.135650 11023 raft_consensus.cc:3055] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
W20260430 07:58:37.136457 10968 consensus_meta.cc:287] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.619s user 0.010s sys 0.001s
I20260430 07:58:37.069012 10993 raft_consensus.cc:3055] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.174911 10972 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.201551 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:37.138098 9859 raft_consensus.cc:2393] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:37.202289 9855 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.203140 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:37.204574 9859 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.028434 11033 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:37.142468 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 125897us
I20260430 07:58:37.175016 11022 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:37.206892 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.029072 11007 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.210088 11007 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:37.210438 10989 consensus_queue.cc:260] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:37.152073 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:37.139771 10995 consensus_meta.cc:287] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.688s user 0.005s sys 0.009s
I20260430 07:58:37.129841 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:37.165334 11035 raft_consensus.cc:3055] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.212764 11035 raft_consensus.cc:740] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:37.213335 10565 raft_consensus.cc:1240] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.168378 9978 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:37.068578 10969 consensus_queue.cc:260] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.137195 11002 raft_consensus.cc:740] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:37.181677 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:37.216097 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:37.144933 10952 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.241374 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:37.192307 9863 raft_consensus.cc:1240] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.136946 10959 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:37.246757 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 326651us
I20260430 07:58:37.248406 9933 raft_consensus.cc:1240] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.180460 10981 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:37.196879 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 290759us
I20260430 07:58:37.193840 11023 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:37.198367 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:37.259092 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:37.259419 9977 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:37.260134 9934 raft_consensus.cc:2376] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2. Re-sending same reply.
I20260430 07:58:37.190999 11027 raft_consensus.cc:740] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:37.170763 11008 raft_consensus.cc:3055] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.239063 10989 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.268977 10008 raft_consensus.cc:1240] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.201730 10993 raft_consensus.cc:740] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:37.147280 10009 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:37.217852 9938 raft_consensus.cc:1240] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: [1.3-1.3]
W20260430 07:58:37.227274 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:37.227123 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 58798.7us
W20260430 07:58:37.236181 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.239499 9862 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.182942 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:37.240025 9935 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.241011 10006 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.250036 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:37.201395 10968 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:37.281409 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:37.254144 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:37.282114 10005 raft_consensus.cc:1275] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:37.282291 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:37.282894 9933 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.242462 10995 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:37.286803 9937 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.285259 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:37.299661 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.286443 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:37.302266 9978 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:37.145212 10563 rpcz_store.cc:276] 0430 07:58:36.110735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:36.112012 (+ 1277us) service_pool.cc:225] Handling call
0430 07:58:36.362299 (+250287us) spinlock_profiling.cc:243] Waited 7.34 ms on lock 0x7b58001f1798. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:36.362740 (+ 441us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:36.854547 (+491807us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:37.126323 (+271776us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b54005901d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:37.126577 (+ 254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":985969664}
I20260430 07:58:37.305485 9978 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:37.280275 10955 consensus_queue.cc:1059] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260430 07:58:37.300978 9977 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:37.313068 9978 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:37.264806 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:37.267480 11008 raft_consensus.cc:740] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:37.269039 11048 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.329726 9903 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:37.331218 9862 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:37.280229 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:37.303812 10969 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.321846 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:37.327610 10005 raft_consensus.cc:1240] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.333006 10004 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:37.265162 10974 consensus_queue.cc:1059] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260430 07:58:37.308157 10918 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:37.323316 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:37.343420 10565 raft_consensus.cc:2393] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:37.261792 10981 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.307281 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:37.325541 10919 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:37.347818 10005 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.343201 11024 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:37.358590 10565 raft_consensus.cc:1240] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.315135 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:37.322412 11021 raft_consensus.cc:2749] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:37.310695 10994 consensus_queue.cc:1059] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260430 07:58:37.337808 10992 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:37.370769 10003 raft_consensus.cc:606] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Received request to transfer leadership to TS d66db415f0f0402fa431349afbf59888
I20260430 07:58:37.371371 10003 raft_consensus.cc:614] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:58:37.310009 11017 consensus_queue.cc:1059] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260430 07:58:37.344976 11004 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:37.372025 9903 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:37.300992 11023 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.352773 10007 raft_consensus.cc:1240] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:37.327970 11043 consensus_queue.cc:1059] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260430 07:58:37.381399 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:37.307579 11002 consensus_queue.cc:260] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.332458 10993 consensus_queue.cc:260] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.390769 10012 raft_consensus.cc:1275] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:37.368019 10944 consensus_queue.cc:1059] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260430 07:58:37.391654 10993 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.394250 11047 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:37.394668 10998 raft_consensus.cc:3055] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.395061 10998 raft_consensus.cc:740] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:37.394838 11047 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.395938 10998 consensus_queue.cc:260] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.397993 11041 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:37.398648 11041 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.350250 11007 consensus_queue.cc:1059] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260430 07:58:37.325811 11035 consensus_queue.cc:260] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:37.409147 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.415642 10994 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:37.416445 10994 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.419119 10994 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:37.375269 11027 consensus_queue.cc:260] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.421171 11027 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.386245 11009 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:37.429454 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:37.397047 11047 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:37.437050 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.408357 11023 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.434087 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:37.461511 10995 raft_consensus.cc:3055] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.463362 10995 raft_consensus.cc:740] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:37.463598 10012 raft_consensus.cc:2393] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:37.440429 10968 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:37.442152 11043 raft_consensus.cc:3055] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.443730 11044 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:37.458312 11034 log.cc:927] Time spent T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260430 07:58:37.444718 11030 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.116s
I20260430 07:58:37.445335 10959 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:37.451937 9936 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:37.380311 11024 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.458410 10565 raft_consensus.cc:2393] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:37.405326 10940 consensus_queue.cc:1059] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260430 07:58:37.465090 9933 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:37.401201 11041 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:37.474409 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.488090 10981 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:37.494423 10012 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:58:37.407274 11008 consensus_queue.cc:260] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.491333 11044 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.466946 11043 raft_consensus.cc:740] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:37.464212 10995 consensus_queue.cc:260] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.485265 10968 raft_consensus.cc:697] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:37.462080 11009 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:37.586818 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:37.600952 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.617964 11018 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:37.618835 11018 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.624311 11018 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:37.557476 10533 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.037s sys 0.080s
I20260430 07:58:37.503660 11006 consensus_queue.cc:1059] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260430 07:58:37.640511 11033 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:37.643930 11024 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:37.644611 10533 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.038s sys 0.080s
I20260430 07:58:37.651703 11033 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.604868 10995 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.603116 10959 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:37.660503 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.672408 10919 raft_consensus.cc:3055] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.672991 10919 raft_consensus.cc:740] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:37.674084 10919 consensus_queue.cc:260] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.676232 10919 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.678404 11009 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:37.678944 11048 raft_consensus.cc:2749] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:37.679456 10563 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:37.682602 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:37.684731 10004 raft_consensus.cc:1240] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:37.686211 10949 consensus_meta.cc:287] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.512s user 0.009s sys 0.003s
W20260430 07:58:37.688431 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.657310 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:37.660832 11006 consensus_queue.cc:1243] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Peer d66db415f0f0402fa431349afbf59888 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260430 07:58:37.659879 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 631287us
I20260430 07:58:37.699741 9935 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:37.701061 11040 raft_consensus.cc:3055] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.702100 10967 raft_consensus.cc:3055] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:37.702518 11040 raft_consensus.cc:740] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:37.702872 10967 raft_consensus.cc:740] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:37.703799 10967 consensus_queue.cc:260] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.707682 10967 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:37.710609 11053 log.cc:927] Time spent T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20260430 07:58:37.710964 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:37.711783 10567 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.690198 10968 consensus_queue.cc:237] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.707819 11035 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.684540 9855 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
W20260430 07:58:37.704891 11054 log.cc:927] Time spent T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
I20260430 07:58:37.723855 11033 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:37.667793 11043 consensus_queue.cc:260] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.732932 11043 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:37.740801 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.670548 10959 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:37.744657 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.718430 11048 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:37.737524 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.751657 11002 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.747454 11020 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:37.751948 11008 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.703632 11040 consensus_queue.cc:260] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.763083 11020 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.775913 10972 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:37.776504 10972 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.778980 10972 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:37.790620 10567 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:37.795318 10012 raft_consensus.cc:606] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:58:37.795897 10012 raft_consensus.cc:614] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260430 07:58:37.798382 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.797078 11040 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:37.794077 11044 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:37.807474 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 485363us
I20260430 07:58:37.809952 9977 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:58:37.822047 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.826529 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:37.827816 10004 raft_consensus.cc:2393] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:37.848891 9977 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:37.821463 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:37.753906 11048 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:37.851490 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.851459 11028 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:37.858667 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:37.847759 11023 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
W20260430 07:58:37.873057 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.860888 10918 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:37.871809 11009 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:37.879007 11009 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.847002 10993 consensus_queue.cc:1059] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260430 07:58:37.886870 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:37.888043 10567 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:37.902714 9977 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:37.903290 9761 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53852 (request call id 88) took 2314 ms. Trace:
I20260430 07:58:37.872539 9855 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:37.863626 9936 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:37.876820 11018 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:37.876364 11023 raft_consensus.cc:697] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:37.918715 10567 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:37.903820 9761 rpcz_store.cc:276] 0430 07:58:35.588938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.597037 (+ 8099us) service_pool.cc:225] Handling call
0430 07:58:37.903198 (+2306161us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:58:35.909740 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:58:35.952142 (+ 42402us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:58:35.952200 (+ 58us) write_op.cc:435] Acquired schema lock
0430 07:58:35.952237 (+ 37us) tablet.cc:662] Decoding operations
0430 07:58:36.020084 (+ 67847us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:58:36.078963 (+ 58879us) write_op.cc:641] Partition lock acquired for write op
0430 07:58:36.094885 (+ 15922us) tablet.cc:685] Acquiring locks for 1 operations
0430 07:58:36.107286 (+ 12401us) tablet.cc:701] Row locks acquired
0430 07:58:36.107330 (+ 44us) write_op.cc:260] PREPARE: finished
0430 07:58:36.107502 (+ 172us) write_op.cc:270] Start()
0430 07:58:36.107793 (+ 291us) write_op.cc:276] Timestamp: P: 1777535916107435 usec, L: 0
0430 07:58:36.107856 (+ 63us) op_driver.cc:348] REPLICATION: starting
0430 07:58:36.275447 (+167591us) log.cc:844] Serialized 535 byte log entry
0430 07:58:37.290601 (+1015154us) op_driver.cc:464] REPLICATION: finished
0430 07:58:37.810538 (+519937us) write_op.cc:301] APPLY: starting
0430 07:58:37.811168 (+ 630us) tablet.cc:1367] starting BulkCheckPresence
0430 07:58:37.811286 (+ 118us) tablet.cc:1370] finished BulkCheckPresence
0430 07:58:37.811424 (+ 138us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:58:37.811727 (+ 303us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:58:37.811968 (+ 241us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 07:58:37.812027 (+ 59us) write_op.cc:312] APPLY: finished
0430 07:58:37.812303 (+ 276us) log.cc:844] Serialized 22 byte log entry
0430 07:58:37.812486 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:37.812564 (+ 78us) write_op.cc:454] Released schema lock
0430 07:58:37.812617 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0430 07:58:37.850888 (+ 38271us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:37.852947 (+ 2059us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":519619,"apply.run_cpu_time_us":2480,"apply.run_wall_time_us":2996,"num_ops":1,"prepare.queue_time_us":261638,"prepare.run_cpu_time_us":51718,"prepare.run_wall_time_us":819275,"raft.queue_time_us":372840,"raft.run_cpu_time_us":74231,"raft.run_wall_time_us":269155,"replication_time_us":1182472,"spinlock_wait_cycles":95044864,"thread_start_us":1103231,"threads_started":4,"wal-append.queue_time_us":538069}]]}
W20260430 07:58:37.933491 9901 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.018s sys 0.029s
W20260430 07:58:37.938097 9901 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s user 0.018s sys 0.029s
I20260430 07:58:37.939867 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:37.936386 9858 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.008s sys 0.043s
W20260430 07:58:37.941512 9858 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s user 0.009s sys 0.043s
I20260430 07:58:37.921370 10959 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:37.936190 11056 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:37.947221 10012 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
W20260430 07:58:37.947415 9904 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.002s sys 0.049s
W20260430 07:58:37.951846 9904 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.002s sys 0.049s
I20260430 07:58:37.944195 10972 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:37.949523 9977 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:37.961984 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:37.961619 9903 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.004s sys 0.044s
I20260430 07:58:37.962894 9933 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:37.908483 11031 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260430 07:58:37.965224 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:37.912612 10533 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:37.973902 9937 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
W20260430 07:58:37.958997 10939 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.016s sys 0.014s
W20260430 07:58:37.975960 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:37.959116 10918 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:37.945721 10993 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:37.930197 11018 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.981595 9937 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:37.976166 10939 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s user 0.017s sys 0.014s
I20260430 07:58:37.959213 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:37.987269 9856 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:37.987571 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:37.960909 10959 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:37.988246 10959 raft_consensus.cc:697] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:37.989152 10959 consensus_queue.cc:237] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:37.994304 10534 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:37.970769 11020 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:37.969534 10928 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.019s sys 0.030s
I20260430 07:58:37.984529 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:37.872022 11028 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.983206 10993 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:37.985688 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:37.995119 9933 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:37.971004 11056 raft_consensus.cc:697] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:37.971626 9903 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.545s user 0.005s sys 0.044s
W20260430 07:58:37.963922 10974 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.008s sys 0.026s
I20260430 07:58:37.997323 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:37.962044 10972 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:38.011399 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.012218 9937 raft_consensus.cc:2393] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:38.003868 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.014142 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:37.982626 10918 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.004611 11020 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.017083 10998 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:38.018894 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.018658 11009 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:38.018797 10008 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
W20260430 07:58:38.006192 10928 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.659s user 0.019s sys 0.030s
I20260430 07:58:37.999347 11048 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:38.018559 10995 raft_consensus.cc:3055] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:38.022029 9933 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:38.023723 9759 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48090 (request call id 188) took 2182 ms. Trace:
W20260430 07:58:38.022644 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 650333us
W20260430 07:58:38.019430 10009 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.025s sys 0.035s
W20260430 07:58:38.035377 9859 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.727s user 0.000s sys 0.055s
W20260430 07:58:38.035746 9978 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.703s user 0.034s sys 0.020s
W20260430 07:58:38.036016 10006 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.023s sys 0.031s
I20260430 07:58:37.978334 11023 consensus_queue.cc:237] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:38.030205 11004 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.012s sys 0.024s
W20260430 07:58:38.024410 10974 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s user 0.008s sys 0.026s
W20260430 07:58:38.041567 9859 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s user 0.000s sys 0.055s
W20260430 07:58:38.041407 10009 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.025s sys 0.035s
I20260430 07:58:37.982079 11052 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
W20260430 07:58:38.041903 10006 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s user 0.024s sys 0.031s
I20260430 07:58:38.042516 9931 raft_consensus.cc:2393] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:58:38.041705 9978 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.709s user 0.034s sys 0.020s
I20260430 07:58:38.041042 11009 raft_consensus.cc:2749] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:58:38.071848 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 758443us
I20260430 07:58:38.072417 11009 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260430 07:58:38.041365 9828 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.028s sys 0.011s
W20260430 07:58:38.065063 10565 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.003s sys 0.027s
I20260430 07:58:38.068434 10003 raft_consensus.cc:1240] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:38.069800 9977 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:38.074316 10011 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:38.073673 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.077248 10993 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:38.100957 9828 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.694s user 0.029s sys 0.012s
W20260430 07:58:38.083575 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.081254 11048 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:38.082049 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.091894 10005 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
W20260430 07:58:38.107136 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 700046us
I20260430 07:58:38.109141 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:38.082146 9934 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.010s sys 0.018s
W20260430 07:58:38.104713 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 110440us
W20260430 07:58:38.110062 9934 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s user 0.011s sys 0.018s
I20260430 07:58:38.079874 9937 raft_consensus.cc:1240] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:38.112651 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:38.116478 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:38.082412 10565 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s user 0.004s sys 0.027s
I20260430 07:58:38.072927 9978 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:38.040127 9759 rpcz_store.cc:276] 0430 07:58:35.841012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:35.841191 (+ 179us) service_pool.cc:225] Handling call
0430 07:58:38.023610 (+2182419us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:58:36.769700 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:58:36.868660 (+ 98960us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:58:36.875546 (+ 6886us) write_op.cc:435] Acquired schema lock
0430 07:58:36.875573 (+ 27us) tablet.cc:662] Decoding operations
0430 07:58:36.901256 (+ 25683us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:58:36.917898 (+ 16642us) write_op.cc:641] Partition lock acquired for write op
0430 07:58:36.922151 (+ 4253us) tablet.cc:685] Acquiring locks for 1 operations
0430 07:58:36.930079 (+ 7928us) tablet.cc:701] Row locks acquired
0430 07:58:36.930102 (+ 23us) write_op.cc:260] PREPARE: finished
0430 07:58:36.930252 (+ 150us) write_op.cc:270] Start()
0430 07:58:36.930379 (+ 127us) write_op.cc:276] Timestamp: P: 1777535916930203 usec, L: 0
0430 07:58:36.930406 (+ 27us) op_driver.cc:348] REPLICATION: starting
0430 07:58:36.931302 (+ 896us) log.cc:844] Serialized 540 byte log entry
0430 07:58:37.961906 (+1030604us) op_driver.cc:464] REPLICATION: finished
0430 07:58:37.979993 (+ 18087us) write_op.cc:301] APPLY: starting
0430 07:58:37.982010 (+ 2017us) tablet.cc:1367] starting BulkCheckPresence
0430 07:58:37.990342 (+ 8332us) tablet.cc:1370] finished BulkCheckPresence
0430 07:58:37.990462 (+ 120us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:58:37.990714 (+ 252us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:58:37.990864 (+ 150us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 07:58:37.990918 (+ 54us) write_op.cc:312] APPLY: finished
0430 07:58:37.991123 (+ 205us) log.cc:844] Serialized 22 byte log entry
0430 07:58:37.991319 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:37.991396 (+ 77us) write_op.cc:454] Released schema lock
0430 07:58:37.991439 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0430 07:58:37.991792 (+ 353us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:37.991846 (+ 54us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":3020,"apply.run_cpu_time_us":1706,"apply.run_wall_time_us":13838,"num_ops":1,"prepare.queue_time_us":840457,"prepare.run_cpu_time_us":3919,"prepare.run_wall_time_us":176238,"replication_time_us":1016822}]]}
I20260430 07:58:38.111965 11009 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:38.139323 11009 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:38.110152 11043 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:58:38.140635 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 67755.9us
W20260430 07:58:38.042693 9929 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.020s sys 0.035s
I20260430 07:58:38.123528 11024 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.158397 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.124086 9937 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:38.159682 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:38.126566 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:38.160586 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.040535 10995 raft_consensus.cc:740] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:38.160884 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.098536 9977 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:38.131505 10003 raft_consensus.cc:1240] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:38.123772 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:38.054134 11004 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s user 0.012s sys 0.024s
W20260430 07:58:38.166994 9828 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.627s (SENT) [suppressed 21 similar messages]
I20260430 07:58:38.168556 9938 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
W20260430 07:58:38.169821 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 87586.4us
I20260430 07:58:38.171131 9977 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:38.140456 11043 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.173022 9978 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:38.173205 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:38.158155 9929 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.021s sys 0.036s
I20260430 07:58:38.174847 9977 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:38.038630 11020 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.174335 10003 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:38.157982 10993 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:38.195271 9978 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:38.176054 10989 consensus_meta.cc:287] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.895s user 0.008s sys 0.010s
I20260430 07:58:38.201373 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:38.202183 9862 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.202649 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:38.203529 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:38.168885 10974 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.070322 10967 consensus_queue.cc:1048] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.112s
I20260430 07:58:38.204135 9861 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.141680 11006 consensus_queue.cc:1059] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260430 07:58:38.169845 11004 raft_consensus.cc:697] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:38.177523 11035 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:38.177145 11031 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:38.177337 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.236124 9978 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:38.237331 11049 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.175526 11048 consensus_queue.cc:1059] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
W20260430 07:58:38.229362 10921 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.765s user 0.009s sys 0.036s
I20260430 07:58:38.203146 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:38.250846 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.167795 10959 consensus_queue.cc:1059] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260430 07:58:38.251677 10993 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.251827 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 999) took 1073 ms. Trace:
I20260430 07:58:38.303754 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 34) took 1039 ms. Trace:
I20260430 07:58:38.192237 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.241068 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.237977 10944 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
W20260430 07:58:38.287484 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 113731us
W20260430 07:58:38.317173 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 414909us
I20260430 07:58:38.157756 11056 consensus_queue.cc:237] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.203532 11019 raft_consensus.cc:2749] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:38.174432 11024 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:38.296914 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:38.280026 10563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.013s sys 0.025s
I20260430 07:58:38.326836 9828 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:38.252528 10919 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
W20260430 07:58:38.330211 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.303753 9903 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:38.317108 11044 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:38.279405 11062 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.320233 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:38.336611 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:38.322067 10921 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.858s user 0.010s sys 0.036s
I20260430 07:58:38.327483 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:38.329712 11035 raft_consensus.cc:697] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:38.253015 10974 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.322319 11049 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.330687 10007 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:38.319587 11031 raft_consensus.cc:697] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:38.393187 9933 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:38.491425 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.491730 9760 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48098 (request call id 190) took 1554 ms. Trace:
I20260430 07:58:38.472906 10919 raft_consensus.cc:697] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:38.424759 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:38.442322 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 115519us
W20260430 07:58:38.488680 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 252507us
I20260430 07:58:38.473404 9938 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
W20260430 07:58:38.449844 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 147863us
I20260430 07:58:38.492671 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.442130 11061 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.326129 11018 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:38.413939 9858 rpcz_store.cc:276] 0430 07:58:37.178466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:37.178656 (+ 190us) service_pool.cc:225] Handling call
0430 07:58:37.179154 (+ 498us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:37.179564 (+ 410us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:37.179588 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:37.180395 (+ 807us) log.cc:844] Serialized 564 byte log entry
0430 07:58:37.942003 (+761608us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:37.942064 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:37.942157 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:38.251526 (+309369us) raft_consensus.cc:1685] finished
0430 07:58:38.251561 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:38.251753 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":761206,"threads_started":1,"wal-append.queue_time_us":1003816}
I20260430 07:58:38.497956 9978 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:38.474661 10993 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.502445 11031 consensus_queue.cc:237] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.503746 10533 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:38.505569 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1008) took 1233 ms. Trace:
I20260430 07:58:38.505642 10533 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:38.506065 9934 rpcz_store.cc:276] 0430 07:58:37.272470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:37.272787 (+ 317us) service_pool.cc:225] Handling call
0430 07:58:37.347728 (+ 74941us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:37.540046 (+192318us) spinlock_profiling.cc:243] Waited 6.31 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a89ec 00007fb5661a85ab 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:37.543953 (+ 3907us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:37.549264 (+ 5311us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:37.549327 (+ 63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:37.553375 (+ 4048us) log.cc:844] Serialized 564 byte log entry
0430 07:58:38.110822 (+557447us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:38.110927 (+ 105us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:38.111023 (+ 96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:38.505362 (+394339us) raft_consensus.cc:1685] finished
0430 07:58:38.505382 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:38.505511 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21775104,"thread_start_us":556953,"threads_started":1,"wal-append.queue_time_us":940806}
I20260430 07:58:38.507648 11080 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:38.507895 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:38.375504 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.509244 11052 raft_consensus.cc:2764] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20260430 07:58:38.461789 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.421348 11059 consensus_queue.cc:1059] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
I20260430 07:58:38.414021 10994 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.511405 10994 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.413805 9859 rpcz_store.cc:276] 0430 07:58:37.214001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:37.214186 (+ 185us) service_pool.cc:225] Handling call
0430 07:58:37.214506 (+ 320us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:37.241428 (+ 26922us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:37.241458 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:37.256119 (+ 14661us) log.cc:844] Serialized 26 byte log entry
0430 07:58:38.069631 (+813512us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:58:38.069860 (+ 229us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:38.069926 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:38.253545 (+183619us) raft_consensus.cc:1685] finished
0430 07:58:38.253576 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:38.253770 (+ 194us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":24672,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":17214,"thread_start_us":754585,"threads_started":1,"wal-append.queue_time_us":967538}
I20260430 07:58:38.503882 10993 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:38.441941 10563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s user 0.014s sys 0.026s
I20260430 07:58:38.419654 11028 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:38.478649 11062 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.514839 11024 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:38.515409 11024 raft_consensus.cc:3055] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:38.515753 11024 raft_consensus.cc:740] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:38.499521 9933 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:38.516239 11062 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:38.516821 11062 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:38.450146 11027 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:38.517870 11027 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:38.518250 11027 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:38.519364 11008 consensus_meta.cc:287] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.756s user 0.020s sys 0.000s
I20260430 07:58:38.520449 9935 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:38.449258 9855 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:38.522133 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.477607 10972 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:38.472364 10967 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:38.458359 10939 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:38.523224 11035 consensus_queue.cc:237] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.488453 9854 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:38.500190 10959 raft_consensus.cc:2804] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:38.472847 11004 consensus_queue.cc:237] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.499538 11061 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.516578 11024 consensus_queue.cc:260] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.523821 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.520632 9759 catalog_manager.cc:5671] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 2 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:38.373291 11067 consensus_queue.cc:1059] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.159s
W20260430 07:58:38.501559 10567 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.013s sys 0.024s
W20260430 07:58:38.558682 10567 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.563s user 0.013s sys 0.024s
I20260430 07:58:38.559392 9759 catalog_manager.cc:5671] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 2 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260430 07:58:38.452190 10968 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.027s sys 0.026s
I20260430 07:58:38.472560 10918 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:38.542582 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 44074us
W20260430 07:58:38.526852 11071 log.cc:927] Time spent T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.204s user 0.002s sys 0.000s
I20260430 07:58:38.571987 9759 catalog_manager.cc:5671] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 1 to 2, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 2 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260430 07:58:38.547180 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 172479us
I20260430 07:58:38.517206 11062 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:38.500833 10921 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.524468 10967 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:38.594337 9934 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:38.595572 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.537060 11043 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:38.571249 10968 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s user 0.028s sys 0.033s
I20260430 07:58:38.584197 9759 catalog_manager.cc:5671] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 2 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:38.562752 9933 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:38.575745 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.492175 9760 rpcz_store.cc:276] 0430 07:58:36.964866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:36.975483 (+ 10617us) service_pool.cc:225] Handling call
0430 07:58:38.491660 (+1516177us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:58:37.176717 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:58:37.177251 (+ 534us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:58:37.177313 (+ 62us) write_op.cc:435] Acquired schema lock
0430 07:58:37.177333 (+ 20us) tablet.cc:662] Decoding operations
0430 07:58:37.177679 (+ 346us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:58:37.177725 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0430 07:58:37.177764 (+ 39us) tablet.cc:685] Acquiring locks for 1 operations
0430 07:58:37.177935 (+ 171us) tablet.cc:701] Row locks acquired
0430 07:58:37.177952 (+ 17us) write_op.cc:260] PREPARE: finished
0430 07:58:37.178589 (+ 637us) write_op.cc:270] Start()
0430 07:58:37.178711 (+ 122us) write_op.cc:276] Timestamp: P: 1777535917178545 usec, L: 0
0430 07:58:37.178727 (+ 16us) op_driver.cc:348] REPLICATION: starting
0430 07:58:37.179309 (+ 582us) log.cc:844] Serialized 535 byte log entry
0430 07:58:37.995996 (+816687us) op_driver.cc:464] REPLICATION: finished
0430 07:58:38.023867 (+ 27871us) write_op.cc:301] APPLY: starting
0430 07:58:38.024534 (+ 667us) tablet.cc:1367] starting BulkCheckPresence
0430 07:58:38.030538 (+ 6004us) tablet.cc:1370] finished BulkCheckPresence
0430 07:58:38.041771 (+ 11233us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:58:38.169121 (+127350us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:58:38.193969 (+ 24848us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0430 07:58:38.203614 (+ 9645us) write_op.cc:312] APPLY: finished
0430 07:58:38.247848 (+ 44234us) log.cc:844] Serialized 22 byte log entry
0430 07:58:38.248368 (+ 520us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:38.248468 (+ 100us) write_op.cc:454] Released schema lock
0430 07:58:38.248513 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0430 07:58:38.248935 (+ 422us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:38.248967 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":6882,"apply.run_cpu_time_us":2869,"apply.run_wall_time_us":227016,"num_ops":1,"prepare.queue_time_us":189,"prepare.run_cpu_time_us":2869,"prepare.run_wall_time_us":8560,"replication_time_us":804295}]]}
W20260430 07:58:38.610118 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 112598us
I20260430 07:58:38.525020 10939 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.630914 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.586269 11062 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:38.631872 9933 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:38.586257 10918 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.598927 10967 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:38.489790 11044 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.637238 10959 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:38.636996 10939 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:38.582897 10919 consensus_queue.cc:237] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.637915 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:38.638993 11081 log.cc:927] Time spent T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
W20260430 07:58:38.639840 11064 log.cc:927] Time spent T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
I20260430 07:58:38.671043 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.641709 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.657197 11040 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:38.673831 11040 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.675627 10006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 790) took 1399 ms. Trace:
I20260430 07:58:38.642581 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.676002 11040 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:38.600553 10994 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:38.656682 10918 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.679534 10918 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:38.663664 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.629909 11024 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:38.680465 11098 consensus_queue.cc:1059] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260430 07:58:38.681124 11060 log.cc:927] Time spent T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.319s user 0.001s sys 0.002s
I20260430 07:58:38.682126 10949 raft_consensus.cc:3055] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:38.682379 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 792) took 1405 ms. Trace:
I20260430 07:58:38.682546 10949 raft_consensus.cc:740] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:38.667101 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:38.668743 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:38.631959 11020 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:38.684636 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.675895 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.597666 10921 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.618793 10944 raft_consensus.cc:697] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:38.686317 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:38.661026 10959 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:38.682799 10009 rpcz_store.cc:276] 0430 07:58:37.276797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:37.311928 (+ 35131us) service_pool.cc:225] Handling call
0430 07:58:37.346859 (+ 34931us) spinlock_profiling.cc:243] Waited 2.48 ms on lock 0x7b58000a2098. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:37.347065 (+ 206us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:37.366530 (+ 19465us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:37.366582 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:37.377247 (+ 10665us) log.cc:844] Serialized 26 byte log entry
0430 07:58:38.050121 (+672874us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:38.055067 (+ 4946us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:38.132482 (+ 77415us) spinlock_profiling.cc:243] Waited 9.92 ms on lock 0x7b5400146c54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:38.132533 (+ 51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:38.680509 (+547976us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:58:38.682170 (+ 1661us) raft_consensus.cc:1685] finished
0430 07:58:38.682192 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:38.682323 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9547,"raft.queue_time_us":9119,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":16172,"spinlock_wait_cycles":56589312,"thread_start_us":638649,"threads_started":1,"wal-append.queue_time_us":834832}
W20260430 07:58:38.679715 11078 log.cc:927] Time spent T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20260430 07:58:38.676064 10006 rpcz_store.cc:276] 0430 07:58:37.276171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:37.307500 (+ 31329us) service_pool.cc:225] Handling call
0430 07:58:37.307833 (+ 333us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:37.316492 (+ 8659us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:37.316512 (+ 20us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:37.326599 (+ 10087us) log.cc:844] Serialized 26 byte log entry
0430 07:58:38.064351 (+737752us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:38.064397 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:38.064464 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:38.675397 (+610933us) raft_consensus.cc:1685] finished
0430 07:58:38.675419 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:38.675541 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8272,"raft.queue_time_us":24386,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":9490,"thread_start_us":708828,"threads_started":1,"wal-append.queue_time_us":985442}
I20260430 07:58:38.693089 9938 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:38.682569 11098 raft_consensus.cc:3055] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:38.694747 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.700814 9935 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:38.696012 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.707376 9933 raft_consensus.cc:2393] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:38.703449 9938 raft_consensus.cc:1240] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:38.710803 9938 raft_consensus.cc:1240] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:38.712718 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:38.696008 10998 consensus_meta.cc:287] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.654s user 0.018s sys 0.006s
I20260430 07:58:38.715101 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.718638 9934 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
W20260430 07:58:38.714607 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 216507us
I20260430 07:58:38.716590 10003 raft_consensus.cc:2393] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:38.690754 10939 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:38.729773 10989 raft_consensus.cc:3055] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:38.730247 10989 raft_consensus.cc:740] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:38.730571 9930 consensus_meta.cc:287] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 1.022s user 0.010s sys 0.014s
I20260430 07:58:38.730926 9930 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.733556 10012 raft_consensus.cc:606] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:58:38.686757 11049 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:38.734210 10012 raft_consensus.cc:614] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:58:38.735285 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.735853 9938 raft_consensus.cc:1275] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:38.736056 10003 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:38.739774 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.694367 10563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54754 (request call id 157) took 1019 ms. Trace:
I20260430 07:58:38.696887 11040 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:38.757345 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.741720 10003 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
W20260430 07:58:38.752333 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 159368us
I20260430 07:58:38.706700 11098 raft_consensus.cc:740] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:38.764834 9903 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:38.711498 10928 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:38.775252 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.776983 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.771476 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.779242 10003 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
W20260430 07:58:38.758247 11053 log.cc:927] Time spent T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260430 07:58:38.743105 11002 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:38.781847 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:38.756671 11006 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.760581 11049 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:38.780931 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:38.762204 9930 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1002) took 1532 ms (client timeout 1911 ms). Trace:
W20260430 07:58:38.802701 11009 consensus_meta.cc:287] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.663s user 0.011s sys 0.000s
W20260430 07:58:38.806444 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.809935 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:38.810302 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.810201 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:38.810632 9861 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:38.810635 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.810883 9859 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.766812 11091 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:38.808843 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 73949.7us
W20260430 07:58:38.806710 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 63567.8us
I20260430 07:58:38.820536 9978 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:38.812079 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.812945 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:38.821563 9859 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.707737 10949 consensus_queue.cc:260] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.821491 9938 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.808938 10003 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:38.824538 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.813169 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:38.803880 11074 log.cc:927] Time spent T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.304s user 0.001s sys 0.002s
I20260430 07:58:38.825155 9938 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.826182 9978 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:38.783398 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1006) took 1567 ms. Trace:
I20260430 07:58:38.783228 10974 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:38.829928 9978 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:38.835873 9978 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:38.833875 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.834470 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.837242 10003 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:38.832763 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:38.838580 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.839389 9934 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.744601 10563 rpcz_store.cc:276] 0430 07:58:37.694401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:37.715454 (+ 21053us) service_pool.cc:225] Handling call
0430 07:58:37.725114 (+ 9660us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:37.725663 (+ 549us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:37.725721 (+ 58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:37.727275 (+ 1554us) log.cc:844] Serialized 564 byte log entry
0430 07:58:38.513694 (+786419us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:38.513738 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:38.513813 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:38.673511 (+159698us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:58:38.683532 (+ 10021us) raft_consensus.cc:1685] finished
0430 07:58:38.683555 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:38.689345 (+ 5790us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5825408,"thread_start_us":785889,"threads_started":1,"wal-append.queue_time_us":848773}
I20260430 07:58:38.840250 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.840456 10919 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:38.840994 9934 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.809695 11069 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.837033 9903 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:58:38.838032 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.841761 11092 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:38.789564 11061 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:38.786311 10928 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.852843 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:38.853104 11092 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.853894 11061 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:38.853976 9857 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.854380 11061 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:38.854839 11061 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:38.855528 11092 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:38.790858 11002 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.847925 9934 raft_consensus.cc:1275] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:38.842144 10994 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:38.846988 9938 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.823256 11031 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260430 07:58:38.857225 9903 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:38.845691 11023 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:38.863909 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.806486 11006 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.851323 11090 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260430 07:58:38.873991 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.860374 9978 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:38.856433 11073 log.cc:927] Time spent T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.274s user 0.000s sys 0.001s
I20260430 07:58:38.821434 11049 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.878468 9903 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:38.807862 9930 rpcz_store.cc:269] 0430 07:58:37.214857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:37.246296 (+ 31439us) service_pool.cc:225] Handling call
0430 07:58:37.646337 (+400041us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:58:38.746826 (+1100489us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48695680}
I20260430 07:58:38.861506 10998 raft_consensus.cc:2955] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:38.875283 10949 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:38.825115 11009 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.856721 11069 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.886864 9934 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:38.831773 11103 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.890036 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.903139 11090 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:38.897257 9861 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:38.908149 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:38.935755 10006 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:38.835263 11091 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.875589 10994 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:38.909507 9933 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:38.915024 10009 raft_consensus.cc:1275] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:38.909029 10972 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:38.916028 11057 raft_consensus.cc:3055] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:38.916584 10921 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:38.918376 11006 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:38.889307 11044 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:38.921382 11049 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:38.922010 11093 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:38.922209 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.942284 10994 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:38.887138 11008 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260430 07:58:38.925331 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.886651 11023 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:38.929476 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:38.930502 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:38.995090 10567 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
W20260430 07:58:38.916057 9936 consensus_meta.cc:287] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.594s user 0.010s sys 0.000s
I20260430 07:58:38.805888 11098 consensus_queue.cc:260] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.934099 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.911644 10560 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:38.911193 11092 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:38.845247 10919 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:38.940954 9903 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:38.940351 11062 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:38.934115 9857 raft_consensus.cc:1240] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20260430 07:58:38.981419 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:38.937776 11090 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:39.014012 11105 log.cc:927] Time spent T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20260430 07:58:38.937686 10939 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:38.909834 11086 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.017674 11086 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:39.018191 11086 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:38.976331 11057 raft_consensus.cc:740] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:38.851770 10944 consensus_queue.cc:237] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
W20260430 07:58:38.998263 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.991711 11008 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:38.938000 10967 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.001603 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:38.833715 9929 rpcz_store.cc:276] 0430 07:58:37.216232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:37.246386 (+ 30154us) service_pool.cc:225] Handling call
0430 07:58:37.354566 (+108180us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:37.405049 (+ 50483us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:37.407555 (+ 2506us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:37.505359 (+ 97804us) log.cc:844] Serialized 564 byte log entry
0430 07:58:38.526037 (+1020678us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:58:38.700866 (+174829us) spinlock_profiling.cc:243] Waited 7.46 ms on lock 0x7b1800247720. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb568e2a85c 00007fb5661c8c0b 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e5242 00007fb568e1c5ab 00007fb568e220c3 00007fb568e22471
0430 07:58:38.715162 (+ 14296us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:38.734363 (+ 19201us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:38.744859 (+ 10496us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:38.756676 (+ 11817us) raft_consensus.cc:1685] finished
0430 07:58:38.762877 (+ 6201us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:38.780839 (+ 17962us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26469632,"thread_start_us":668091,"threads_started":1,"wal-append.queue_time_us":674161}
I20260430 07:58:39.003456 11002 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.003589 11098 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:39.004069 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.002146 11092 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:38.938962 11061 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:38.985560 11044 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:38.940953 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:39.002799 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 61933.3us
I20260430 07:58:39.006846 9934 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:38.999380 9936 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:39.061435 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:38.991818 11093 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.060179 11035 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:39.060635 11094 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260430 07:58:39.065665 11035 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:39.066982 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:38.999483 11023 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:39.037878 11008 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:39.009164 11095 log.cc:927] Time spent T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260430 07:58:39.010775 10959 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.005039 10919 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.089737 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:39.092038 11091 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:39.093824 11027 consensus_meta.cc:287] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.575s user 0.011s sys 0.000s
I20260430 07:58:39.094206 11069 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.094439 11027 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.096658 11061 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:39.067629 10934 raft_consensus.cc:2804] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.077634 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 715) took 1047 ms. Trace:
I20260430 07:58:39.098779 11062 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.098376 9936 rpcz_store.cc:276] 0430 07:58:38.029974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:38.142255 (+112281us) service_pool.cc:225] Handling call
0430 07:58:39.074564 (+932309us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:39.021607 9978 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:39.058750 11092 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:39.102310 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 80786.4us
I20260430 07:58:38.938279 11103 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.104156 9933 raft_consensus.cc:1240] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20260430 07:58:38.978367 10972 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.006161 11024 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:39.013055 10995 consensus_queue.cc:260] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.046710 9855 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:39.057899 10928 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.057255 11057 consensus_queue.cc:260] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.064142 9903 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:58:39.123682 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 59581.2us
I20260430 07:58:39.068153 10009 raft_consensus.cc:1240] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:38.994239 9930 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
W20260430 07:58:39.082675 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.128244 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:39.129083 10006 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:39.080972 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:39.087409 10959 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.088169 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:39.136067 9857 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.106861 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:39.106998 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:39.140753 9858 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.140847 9862 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.106366 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.143286 9860 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:39.144629 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.057390 9931 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:39.116312 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.067744 11059 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:39.119304 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 296366us
I20260430 07:58:39.158762 9861 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:39.153167 9860 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:39.067327 11093 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.133029 9933 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:39.160658 10534 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:39.100955 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:39.097193 11027 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:39.075914 10949 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260430 07:58:39.131248 9934 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:39.119788 11101 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.162189 9978 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:39.163465 9903 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:39.163893 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:39.117630 11062 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.112126 9932 raft_consensus.cc:2368] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:39.119055 11024 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:39.178249 9978 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:39.193349 9978 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:39.178280 10006 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:39.175266 9829 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.508s user 0.021s sys 0.015s
I20260430 07:58:39.079313 11044 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.122290 11009 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.214989 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:39.190408 9861 raft_consensus.cc:1240] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:39.149587 11061 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.179000 9858 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:39.194659 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:39.121923 11031 consensus_queue.cc:1059] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260430 07:58:39.218247 10967 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.194411 11024 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:39.158972 9929 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:39.184019 11052 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.224701 9862 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:39.245630 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.245954 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:39.246356 9862 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:39.247200 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.248203 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.248836 11085 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.249320 11085 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:39.249666 11085 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.232618 9936 raft_consensus.cc:2393] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:39.188159 11043 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.228523 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:39.210438 11035 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.230867 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:39.260838 10009 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.232656 9934 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:39.261379 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.184937 11093 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.234686 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:39.226351 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:39.238857 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:39.221803 10928 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.243515 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
W20260430 07:58:39.272027 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.272106 10567 raft_consensus.cc:2393] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:39.182268 11059 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.203851 10959 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.202766 11006 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.238862 11050 raft_consensus.cc:3055] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
W20260430 07:58:39.232249 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.219475 11103 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.284289 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:55676 (request call id 3) took 1346 ms. Trace:
I20260430 07:58:39.286155 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.284682 10012 rpcz_store.cc:276] 0430 07:58:37.937694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:37.937927 (+ 233us) service_pool.cc:225] Handling call
0430 07:58:39.284088 (+1346161us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b54002df0d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb56927dc33 00007fb56e259cca 00007fb5677d2934 00007fb5677d28c2 00007fb5677d27f1 00007fb5677d276c 00007fb5677d1a22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:39.284205 (+ 117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1796027264}
I20260430 07:58:39.256409 11091 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.259506 11043 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:39.309502 11043 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.310009 10994 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.311015 11050 raft_consensus.cc:740] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:39.312423 11050 consensus_queue.cc:260] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.227015 11061 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:39.314162 11059 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:39.321753 11061 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.321954 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:39.266492 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.221166 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:39.244989 11052 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:39.236218 11057 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:39.192409 11062 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.245684 11079 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.277257 10928 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.274690 9828 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:58:39.353824 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 79160.6us
I20260430 07:58:39.245107 11009 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:39.283526 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.356109 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:39.283658 11044 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:39.224882 10006 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:39.289865 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:39.361721 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:39.363373 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:39.364984 10006 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:39.367759 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:39.298933 9904 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:39.276688 10959 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:39.305816 9934 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:39.306421 10972 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:39.306927 11090 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.307343 11006 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.266152 11093 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:39.307986 10533 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:39.284732 9857 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:39.308770 11091 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.403115 9863 raft_consensus.cc:1275] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260430 07:58:39.329123 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.329988 10012 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:39.326778 11020 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:39.340010 11082 log.cc:927] Time spent T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.796s user 0.002s sys 0.000s
W20260430 07:58:39.349619 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.304939 11085 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.389312 10959 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.338279 11052 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.376524 10994 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.418722 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.420228 9863 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:39.405479 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:39.367635 11009 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.374065 10006 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "d66db415f0f0402fa431349afbf59888"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:58:39.324899 11024 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.434582 11006 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.436156 11024 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.437254 9938 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:39.379344 10009 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:39.366055 11050 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:39.410403 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 102444us
I20260430 07:58:39.362282 11098 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:39.381210 11079 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.397692 11093 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.287571 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.390734 11004 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:39.412135 9903 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:39.412504 10534 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:58:39.375828 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 76960.9us
I20260430 07:58:39.468628 9904 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:39.466233 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 60554.3us
I20260430 07:58:39.425850 10006 raft_consensus.cc:606] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Received request to transfer leadership to TS d66db415f0f0402fa431349afbf59888
I20260430 07:58:39.426684 11112 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.422763 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.427914 10934 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:39.429432 9930 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.432054 11009 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.414076 11020 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.433012 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.434203 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.374998 11002 consensus_queue.cc:1059] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260430 07:58:39.413331 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:39.456996 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.392721 11096 consensus_queue.cc:1059] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260430 07:58:39.431744 11008 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.522583 11114 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:39.523819 11114 raft_consensus.cc:697] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:39.523710 10959 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.525485 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.526271 9861 raft_consensus.cc:2393] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:39.525952 11061 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.488762 10533 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:58:39.530123 9932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 717) took 1487 ms (client timeout 1627 ms). Trace:
I20260430 07:58:39.529873 11043 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:39.530568 9932 rpcz_store.cc:269] 0430 07:58:38.042506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:38.057378 (+ 14872us) service_pool.cc:225] Handling call
0430 07:58:39.529874 (+1472496us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b54003232d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:39.530057 (+ 183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1378461056}
I20260430 07:58:39.532114 11043 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.533664 9931 raft_consensus.cc:2393] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:39.534327 9936 raft_consensus.cc:1240] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:39.534461 10919 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:39.535835 11086 consensus_meta.cc:287] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.517s user 0.005s sys 0.007s
I20260430 07:58:39.536253 11086 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.454463 10989 consensus_queue.cc:260] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.471844 11107 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.472059 9903 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:39.485142 11092 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.497668 10972 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:39.470259 10006 raft_consensus.cc:614] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260430 07:58:39.473186 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 60767.8us
I20260430 07:58:39.478209 11101 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:39.572395 11101 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.546797 11107 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.572541 11085 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.482244 11079 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.570226 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.575009 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.575457 9861 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:39.576237 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.483412 9857 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:39.483737 11084 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:39.485918 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 718) took 1343 ms. Trace:
I20260430 07:58:39.490108 10934 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:39.490249 11117 log.cc:927] Time spent T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20260430 07:58:39.491312 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1251) took 1419 ms. Trace:
I20260430 07:58:39.491799 11112 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.447119 11044 consensus_queue.cc:1048] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260430 07:58:39.493986 11031 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.544330 10563 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:39.502521 9863 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
W20260430 07:58:39.505501 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:39.530323 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 65624.3us
W20260430 07:58:39.444880 9854 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.016s sys 0.053s
W20260430 07:58:39.541829 11097 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.651s user 0.007s sys 0.000s
I20260430 07:58:39.525097 11114 consensus_queue.cc:237] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.534348 11023 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.542209 11002 raft_consensus.cc:3055] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:39.471170 11006 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
W20260430 07:58:39.549723 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 77692.6us
I20260430 07:58:39.559393 11020 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.585498 11052 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.554397 11090 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.579020 10534 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:39.617048 11084 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.617480 11084 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:39.617843 11084 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.596969 10989 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:39.552796 10959 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.583600 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.543404 11086 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.523985 11008 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:39.581434 9937 rpcz_store.cc:276] 0430 07:58:38.142643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:38.160615 (+ 17972us) service_pool.cc:225] Handling call
0430 07:58:39.485708 (+1325093us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001ee954. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:39.485856 (+ 148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2739814272}
I20260430 07:58:39.553620 11103 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:39.589792 10011 rpcz_store.cc:276] 0430 07:58:38.071998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:38.082037 (+ 10039us) service_pool.cc:225] Handling call
0430 07:58:38.084507 (+ 2470us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:38.091597 (+ 7090us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:38.092636 (+ 1039us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:38.101325 (+ 8689us) log.cc:844] Serialized 26 byte log entry
0430 07:58:38.194425 (+ 93100us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:38.194486 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:38.194571 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:39.287407 (+1092836us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:58:39.491013 (+203606us) raft_consensus.cc:1685] finished
0430 07:58:39.491035 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:39.491253 (+ 218us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":886,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":134,"spinlock_wait_cycles":10950528,"thread_start_us":88329,"threads_started":1,"wal-append.queue_time_us":310564}
I20260430 07:58:39.614877 11002 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:39.553934 11062 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.573662 11009 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:39.639145 11009 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:39.548077 11096 raft_consensus.cc:3055] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:39.641762 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:39.553746 10919 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.502166 9904 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:39.600914 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.551643 11092 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.605631 10533 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:39.655283 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 49802us
I20260430 07:58:39.655090 11002 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.656682 11002 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:39.529140 11061 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:39.657441 11077 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:39.657912 11020 consensus_queue.cc:1059] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260430 07:58:39.611842 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.658622 11077 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:58:39.546535 10995 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:39.613796 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:39.665705 11062 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:58:39.607261 9854 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.718s user 0.016s sys 0.053s
I20260430 07:58:39.666462 11062 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.667344 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 41) took 1140 ms. Trace:
I20260430 07:58:39.592819 10928 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.581959 9858 raft_consensus.cc:2393] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:39.647284 11103 raft_consensus.cc:697] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:39.619539 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:39.669631 9862 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:39.620355 9977 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.851s user 0.044s sys 0.110s
I20260430 07:58:39.628099 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:39.671525 10009 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.670413 10567 raft_consensus.cc:2393] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:39.660655 9903 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:58:39.673329 9977 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.904s user 0.044s sys 0.110s
I20260430 07:58:39.630852 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:39.542212 11115 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.171s
I20260430 07:58:39.616057 11098 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.637161 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:39.643359 10012 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.635519 11091 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.644832 11009 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.629411 11006 raft_consensus.cc:697] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:39.629967 10003 raft_consensus.cc:1240] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:39.677397 9855 raft_consensus.cc:2393] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:39.678228 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:39.691001 10008 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:39.650113 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 148008us
I20260430 07:58:39.603883 11031 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.645108 11096 raft_consensus.cc:740] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:58:39.689085 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:39.673835 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 904543us
I20260430 07:58:39.670678 11024 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:39.675215 11061 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:39.699311 9904 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:39.706696 9978 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.017s sys 0.035s
W20260430 07:58:39.708266 9978 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s user 0.018s sys 0.036s
I20260430 07:58:39.714184 10012 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:39.717247 9862 raft_consensus.cc:1275] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:39.717679 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:39.718704 9858 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.689919 11062 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.720649 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:39.693571 11107 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:39.705931 11049 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.018s sys 0.026s
I20260430 07:58:39.728770 9937 raft_consensus.cc:1275] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:39.750706 9933 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:39.726722 9862 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:39.722864 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:39.683113 11098 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.742733 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:39.757161 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:39.757812 9938 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.763633 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:39.764505 9862 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.764972 9937 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:39.766251 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:39.766497 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:39.767253 9978 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:39.721930 9856 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.853s user 0.000s sys 0.025s
I20260430 07:58:39.748293 9904 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:39.752682 9861 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.753388 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:39.704322 11108 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.018s sys 0.035s
I20260430 07:58:39.772506 9859 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:39.770321 9935 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.830s user 0.006s sys 0.017s
I20260430 07:58:39.741335 10009 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:58:39.721482 11034 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.011s sys 0.013s
W20260430 07:58:39.775643 9935 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.835s user 0.007s sys 0.017s
W20260430 07:58:39.774798 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:39.776902 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.723541 11043 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260430 07:58:39.752249 11049 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s user 0.019s sys 0.026s
W20260430 07:58:39.781014 10944 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.000s sys 0.031s
I20260430 07:58:39.770061 9863 raft_consensus.cc:2955] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:39.783661 9978 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:39.785413 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 59) took 1278 ms. Trace:
W20260430 07:58:39.774808 11018 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.012s sys 0.016s
W20260430 07:58:39.775449 11108 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s user 0.019s sys 0.035s
I20260430 07:58:39.792271 9937 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.794873 11090 raft_consensus.cc:2804] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
W20260430 07:58:39.780746 11034 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s user 0.013s sys 0.013s
I20260430 07:58:39.806587 11023 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:39.772472 11086 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.803526 9935 rpcz_store.cc:276] 0430 07:58:38.507397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:38.708000 (+200603us) service_pool.cc:225] Handling call
0430 07:58:38.708955 (+ 955us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:38.709190 (+ 235us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:58:38.935817 (+226627us) log.cc:844] Serialized 14 byte log entry
0430 07:58:39.785033 (+849216us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:39.785091 (+ 58us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:39.785146 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:39.785217 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:39.785361 (+ 144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8581,"prepare.queue_time_us":469257,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":12534,"thread_start_us":1031717,"threads_started":2}
W20260430 07:58:39.807173 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 389305us
I20260430 07:58:39.775583 11091 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:39.833640 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:39.788650 10994 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.745872 11024 consensus_queue.cc:1059] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260430 07:58:39.786547 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:39.785857 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:39.784704 11059 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260430 07:58:39.772743 9856 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.904s user 0.000s sys 0.026s
I20260430 07:58:39.702085 11103 consensus_queue.cc:237] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:39.764505 11062 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:39.862903 10004 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.013s sys 0.021s
W20260430 07:58:39.983362 10004 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.713s user 0.014s sys 0.021s
W20260430 07:58:39.866719 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:39.984879 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:39.862385 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 106820us
W20260430 07:58:39.838385 11018 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.612s user 0.013s sys 0.017s
I20260430 07:58:39.873850 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:39.993320 9935 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.935360 11092 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:39.668138 9854 rpcz_store.cc:276] 0430 07:58:38.527383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:38.530627 (+ 3244us) service_pool.cc:225] Handling call
0430 07:58:38.594334 (+ 63707us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:38.618096 (+ 23762us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:58:38.844583 (+226487us) log.cc:844] Serialized 14 byte log entry
0430 07:58:39.666621 (+822038us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:39.666706 (+ 85us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:39.666779 (+ 73us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:39.666887 (+ 108us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:39.667063 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":371447,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":178499,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":124,"thread_start_us":898527,"threads_started":2}
W20260430 07:58:39.841862 10944 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s user 0.000s sys 0.032s
I20260430 07:58:39.892032 9828 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:58:39.842278 11052 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:39.870054 11116 log.cc:927] Time spent T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.200s user 0.000s sys 0.002s
I20260430 07:58:39.935735 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:40.006842 9903 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:40.014494 9934 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:39.951745 9862 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:39.952282 9938 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:40.016682 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:40.017437 9938 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:39.956466 9858 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:39.783058 11006 consensus_queue.cc:237] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.864825 11112 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:39.960373 10567 raft_consensus.cc:2955] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:39.993847 10563 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:40.031647 9936 raft_consensus.cc:1275] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:39.972858 9855 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 3
I20260430 07:58:40.033834 11108 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:39.935194 11023 consensus_queue.cc:1048] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260430 07:58:39.935204 10959 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
W20260430 07:58:39.908653 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:39.999698 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 107749us
I20260430 07:58:39.985559 9929 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:40.013990 10934 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.024717 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:40.025759 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:40.087460 9858 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:39.934794 11086 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:39.805320 11096 consensus_queue.cc:260] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.024390 11031 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:40.035584 9903 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:40.077721 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:40.034341 11108 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:39.873849 11090 raft_consensus.cc:697] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:40.046861 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:40.093843 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:40.086678 9934 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:39.955159 10994 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.155817 11090 consensus_queue.cc:237] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.158757 10934 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:40.130565 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:40.138927 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:40.161577 9936 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:40.150926 11091 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.096551 11114 consensus_queue.cc:1059] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.059s
I20260430 07:58:40.168681 9931 raft_consensus.cc:1275] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:40.154873 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:40.209949 9934 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:40.034802 11118 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:40.074376 10972 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:39.991649 11093 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:40.162003 11121 log.cc:927] Time spent T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.200s user 0.000s sys 0.003s
I20260430 07:58:39.997113 11062 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.097241 10959 raft_consensus.cc:697] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:40.170293 11043 raft_consensus.cc:3055] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:40.234637 10928 consensus_queue.cc:1059] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:58:40.108808 11108 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:40.252287 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:40.249375 11043 raft_consensus.cc:740] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:40.227952 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:40.273765 11043 consensus_queue.cc:260] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.276994 11043 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:40.014714 11092 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:40.290729 10959 consensus_queue.cc:237] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.228420 11118 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.293874 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:40.330382 11052 raft_consensus.cc:3055] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:40.084327 11061 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:40.355178 9935 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:40.346650 9828 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:40.354663 11052 raft_consensus.cc:740] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:40.355077 10994 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:40.254415 10972 raft_consensus.cc:697] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:58:40.378573 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 342912us
I20260430 07:58:40.358748 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:40.359283 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:40.381881 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 747) took 1257 ms. Trace:
I20260430 07:58:40.382741 9933 rpcz_store.cc:276] 0430 07:58:39.123706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.161806 (+ 38100us) service_pool.cc:225] Handling call
0430 07:58:40.380154 (+1218348us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b5400348d54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:40.380413 (+ 259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1211796224}
I20260430 07:58:40.387279 9857 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:40.387156 11061 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.392530 10934 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:40.407477 11061 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:40.391440 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:40.409184 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1273) took 1142 ms. Trace:
I20260430 07:58:40.409624 10005 rpcz_store.cc:276] 0430 07:58:39.267288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.272860 (+ 5572us) service_pool.cc:225] Handling call
0430 07:58:39.280659 (+ 7799us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:39.281208 (+ 549us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:39.281250 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:39.321356 (+ 40106us) log.cc:844] Serialized 26 byte log entry
0430 07:58:39.773704 (+452348us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:39.773747 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:39.773841 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:40.408963 (+635122us) raft_consensus.cc:1685] finished
0430 07:58:40.408983 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:40.409126 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":103,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":393304,"threads_started":1,"wal-append.queue_time_us":946040}
I20260430 07:58:40.399303 11086 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:40.391868 11044 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:40.411180 9936 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:40.412264 9938 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:40.392838 9828 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:40.387735 11107 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:40.361922 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:40.428344 10008 consensus_meta.cc:287] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.733s user 0.015s sys 0.037s
I20260430 07:58:40.438530 10008 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:58:40.418267 9859 consensus_meta.cc:287] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.645s user 0.005s sys 0.045s
I20260430 07:58:40.444018 9859 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:40.429811 10010 consensus_meta.cc:287] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.755s user 0.025s sys 0.028s
I20260430 07:58:40.445559 10010 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:40.445856 11049 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:40.419821 9932 raft_consensus.cc:2393] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:40.429616 11044 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:40.419878 11052 consensus_queue.cc:260] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:40.448215 11044 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:40.448848 11052 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:40.427315 10972 consensus_queue.cc:237] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.433596 9828 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:40.354229 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 120889us
W20260430 07:58:40.468192 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:40.469578 11114 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:40.430362 11092 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:40.470323 11114 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:40.465687 11049 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:40.470141 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:40.454262 9937 consensus_meta.cc:287] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.599s user 0.011s sys 0.004s
I20260430 07:58:40.416621 11098 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:40.426189 11062 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:40.484247 9937 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:40.489305 9904 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.083s sys 0.157s
W20260430 07:58:40.489894 9904 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.083s sys 0.157s
I20260430 07:58:40.480020 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:40.492935 10011 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.033s sys 0.124s
W20260430 07:58:40.493700 10011 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s user 0.034s sys 0.125s
I20260430 07:58:40.476198 11096 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:40.490257 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 706350us
I20260430 07:58:40.514029 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:40.515043 10005 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:40.514082 9904 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:40.518688 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:40.483940 11049 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:40.537793 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:40.491396 11091 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:40.466771 11108 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:40.537648 10934 raft_consensus.cc:697] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:40.548748 10934 consensus_queue.cc:237] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:40.576148 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 62170.9us
I20260430 07:58:40.578486 9904 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:40.539458 11049 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:40.537802 11008 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:40.538396 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:40.595261 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:40.597146 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:40.597679 10008 raft_consensus.cc:1240] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:40.535912 11118 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:40.594730 11061 raft_consensus.cc:2749] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260430 07:58:40.608918 10995 consensus_meta.cc:287] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.947s user 0.020s sys 0.005s
I20260430 07:58:40.609413 11115 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:40.594959 11024 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:40.613078 9828 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: d66db415f0f0402fa431349afbf59888
W20260430 07:58:40.615111 11084 consensus_meta.cc:287] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.997s user 0.012s sys 0.019s
W20260430 07:58:40.615996 11101 consensus_meta.cc:287] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 1.043s user 0.014s sys 0.001s
W20260430 07:58:40.618076 9930 consensus_meta.cc:287] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 1.080s user 0.015s sys 0.001s
I20260430 07:58:40.618554 11108 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:40.567739 11098 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:40.594935 11002 consensus_meta.cc:287] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.935s user 0.007s sys 0.013s
I20260430 07:58:40.580058 11092 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:40.644071 9863 consensus_meta.cc:287] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.833s user 0.012s sys 0.020s
I20260430 07:58:40.601884 11093 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:40.592705 11059 raft_consensus.cc:2749] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260430 07:58:40.630131 9861 consensus_meta.cc:287] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.855s user 0.014s sys 0.007s
I20260430 07:58:40.593026 9929 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:40.648792 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:40.678339 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:40.627893 9975 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.014s sys 0.025s
W20260430 07:58:40.692217 9977 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.016s sys 0.025s
W20260430 07:58:40.698477 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 54486.4us
I20260430 07:58:40.628891 11107 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:40.702622 9861 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:40.718736 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:40.621431 10563 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
W20260430 07:58:40.727931 9977 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.895s user 0.016s sys 0.027s
W20260430 07:58:40.728366 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 895244us
I20260430 07:58:40.728662 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1034) took 1006 ms. Trace:
I20260430 07:58:40.761780 9861 rpcz_store.cc:276] 0430 07:58:39.722471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.722615 (+ 144us) service_pool.cc:225] Handling call
0430 07:58:40.728596 (+1005981us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:40.724720 9975 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.855s user 0.016s sys 0.025s
I20260430 07:58:40.600726 11008 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:40.700349 10989 consensus_meta.cc:287] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 1.042s user 0.004s sys 0.009s
I20260430 07:58:40.628865 9930 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:40.698518 11122 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:40.750634 9828 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:58:40.729717 9856 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.027s sys 0.042s
W20260430 07:58:40.725162 10998 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.011s sys 0.008s
W20260430 07:58:40.734910 11018 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.022s sys 0.049s
I20260430 07:58:40.628336 11101 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.629360 11024 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.629792 11115 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:40.776130 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:40.766510 9977 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:40.629205 11084 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:40.699958 11098 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.764178 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:40.821780 10008 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:40.750172 11107 raft_consensus.cc:697] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:40.716882 11092 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.746743 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:40.771404 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:40.767385 10995 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:58:40.812978 10004 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.000s sys 0.023s
I20260430 07:58:40.816166 9937 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260430 07:58:40.765760 9828 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:40.818077 9977 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
W20260430 07:58:40.791553 11004 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.010s sys 0.015s
W20260430 07:58:40.769312 9856 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s user 0.028s sys 0.042s
W20260430 07:58:40.794375 10998 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.901s user 0.012s sys 0.008s
I20260430 07:58:40.827179 10010 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:40.795014 11091 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:40.767932 11114 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:40.769104 11093 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:40.849669 9977 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:40.794234 11018 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.725s user 0.026s sys 0.053s
I20260430 07:58:40.791683 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 65) took 1461 ms. Trace:
I20260430 07:58:40.853883 10919 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260430 07:58:40.822813 10989 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:40.845746 11118 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:40.875370 11118 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:40.875831 11118 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:40.826275 10003 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:40.878762 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:40.879848 9937 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:40.882189 10567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54754 (request call id 166) took 1011 ms. Trace:
W20260430 07:58:40.830946 10004 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.847s user 0.000s sys 0.023s
I20260430 07:58:40.884106 9860 raft_consensus.cc:1240] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:40.886509 10004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 826) took 1620 ms. Trace:
I20260430 07:58:40.886987 10004 rpcz_store.cc:276] 0430 07:58:39.266564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.266933 (+ 369us) service_pool.cc:225] Handling call
0430 07:58:39.269992 (+ 3059us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:39.270187 (+ 195us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:39.983788 (+713601us) log.cc:844] Serialized 14 byte log entry
0430 07:58:40.886054 (+902266us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:40.886131 (+ 77us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:40.886189 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:40.886278 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:40.886442 (+ 164us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1269478,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"thread_start_us":1612897,"threads_started":2}
I20260430 07:58:40.890517 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:40.891018 9978 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.060s user 0.007s sys 0.042s
W20260430 07:58:40.891469 9978 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.061s user 0.007s sys 0.042s
I20260430 07:58:40.863202 9977 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:40.853951 10994 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260430 07:58:40.851320 9932 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:40.892788 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:40.867897 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:40.896481 9938 raft_consensus.cc:2393] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:40.868867 11018 raft_consensus.cc:3055] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Stepping down as leader of term 2
W20260430 07:58:40.869750 11135 log.cc:927] Time spent T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
W20260430 07:58:40.832733 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 67056.8us
W20260430 07:58:40.838361 11004 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.603s user 0.011s sys 0.015s
I20260430 07:58:40.871261 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1282) took 1233 ms. Trace:
I20260430 07:58:40.872475 10998 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:40.853778 11086 consensus_queue.cc:1048] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260430 07:58:40.873257 11035 consensus_queue.cc:1048] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260430 07:58:40.873593 9855 consensus_meta.cc:287] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.840s user 0.000s sys 0.032s
W20260430 07:58:40.825407 11124 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.783s user 0.010s sys 0.017s
I20260430 07:58:40.861763 9930 rpcz_store.cc:276] 0430 07:58:39.329881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.338020 (+ 8139us) service_pool.cc:225] Handling call
0430 07:58:40.769347 (+1431327us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:40.899518 11043 consensus_meta.cc:287] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.564s user 0.010s sys 0.003s
I20260430 07:58:40.900098 10011 rpcz_store.cc:276] 0430 07:58:39.639198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.652243 (+ 13045us) service_pool.cc:225] Handling call
0430 07:58:39.686310 (+ 34067us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:39.686536 (+ 226us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:39.686561 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:39.692778 (+ 6217us) log.cc:844] Serialized 26 byte log entry
0430 07:58:40.495007 (+802229us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:40.495067 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:40.495166 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:40.871040 (+375874us) raft_consensus.cc:1685] finished
0430 07:58:40.871068 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:40.871193 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":760820,"threads_started":1,"wal-append.queue_time_us":1039276}
W20260430 07:58:40.907843 9828 leader_election.cc:336] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.505s (SENT)
I20260430 07:58:40.919137 9828 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:40.838447 11062 raft_consensus.cc:2764] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260430 07:58:40.877316 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:40.922114 9828 leader_election.cc:336] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.549s (SENT)
I20260430 07:58:40.861155 11091 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.871280 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:44006 (request call id 441) took 2284 ms. Trace:
I20260430 07:58:40.796106 11122 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:40.891803 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 1.08432e+06us
W20260430 07:58:40.898203 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:40.915033 9936 raft_consensus.cc:1275] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260430 07:58:40.863494 11085 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.633s user 0.002s sys 0.027s
I20260430 07:58:40.912833 9977 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:40.917361 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1232) took 1728 ms. Trace:
I20260430 07:58:40.936219 9860 rpcz_store.cc:276] 0430 07:58:39.188942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.189122 (+ 180us) service_pool.cc:225] Handling call
0430 07:58:40.883873 (+1694751us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:40.884617 (+ 744us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:40.917138 (+ 32521us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54002706d0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:40.917304 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3830013312}
I20260430 07:58:40.936817 9978 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:40.914856 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:40.923243 9828 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:40.888972 11006 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.013s sys 0.006s
I20260430 07:58:40.864909 11093 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.899075 11018 raft_consensus.cc:740] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
W20260430 07:58:40.953588 9828 leader_election.cc:336] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.627s (SENT)
I20260430 07:58:40.954430 9828 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:40.903990 11140 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260430 07:58:40.922551 11124 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.897s user 0.010s sys 0.017s
I20260430 07:58:40.909667 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 43) took 2133 ms. Trace:
I20260430 07:58:40.866111 10995 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.923014 11108 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:40.959672 9978 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:40.929044 10998 raft_consensus.cc:697] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:40.935802 11085 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.713s user 0.003s sys 0.028s
W20260430 07:58:40.952044 11006 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s user 0.014s sys 0.006s
I20260430 07:58:40.937958 9929 raft_consensus.cc:2393] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:40.960320 10008 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:40.943109 11101 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:40.929527 11141 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:40.928755 11044 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:40.977113 9978 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:40.979329 11086 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:40.979537 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:40.958931 11140 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:40.989157 9936 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:40.979537 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:40.913946 11107 consensus_queue.cc:237] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:40.959062 11130 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:40.971251 10972 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.007s sys 0.025s
I20260430 07:58:40.985203 9978 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:40.943464 11062 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260430 07:58:40.997617 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:40.993893 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 396739us
W20260430 07:58:40.975225 10959 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.008s sys 0.026s
I20260430 07:58:40.999279 9861 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:40.999598 10959 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s user 0.009s sys 0.027s
I20260430 07:58:40.999933 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:40.915006 9855 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:40.984176 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 387209us
I20260430 07:58:40.992070 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:40.986045 9938 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:41.003711 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:41.006659 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:41.007654 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:41.008334 9858 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:41.009660 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:41.010387 9858 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:41.011417 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:41.012084 9858 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:40.977620 11008 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:40.990316 9860 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:40.981019 11035 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:41.016018 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:41.020862 10010 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:40.929437 10007 rpcz_store.cc:276] 0430 07:58:38.587161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:38.603676 (+ 16515us) service_pool.cc:225] Handling call
0430 07:58:40.870235 (+2266559us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b54003903d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ed470 00007fb56928c31a 00007fb56e25b11e 00007fb5677d7437 00007fb5677d73c2 00007fb5677d72f1 00007fb5677d726c 00007fb5677d6522 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:40.871198 (+ 963us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4280578816}
I20260430 07:58:40.971561 11139 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:41.016943 9978 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:41.017549 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:41.018293 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:40.982136 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:40.981158 11024 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:41.018729 11009 consensus_meta.cc:287] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 1.324s user 0.013s sys 0.008s
W20260430 07:58:40.996093 10972 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s user 0.008s sys 0.025s
I20260430 07:58:41.020254 9903 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:40.985234 11115 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:41.022989 9858 raft_consensus.cc:1275] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:41.000196 10959 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:41.072775 11093 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:41.072388 10959 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.026316 9978 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:40.993572 11140 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.071017 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:41.075112 9858 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.027710 9862 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.022112 11035 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:40.882706 10567 rpcz_store.cc:276] 0430 07:58:39.933940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.958771 (+ 24831us) service_pool.cc:225] Handling call
0430 07:58:39.959682 (+ 911us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:39.960148 (+ 466us) raft_consensus.cc:1534] Early marking committed up to index 3
0430 07:58:40.464180 (+504032us) log.cc:844] Serialized 14 byte log entry
0430 07:58:40.881791 (+417611us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:40.881842 (+ 51us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:58:40.881881 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:40.882025 (+ 144us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:40.882137 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1214,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":11741,"thread_start_us":417131,"threads_started":1}
I20260430 07:58:41.077543 11035 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.078390 10959 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:40.999969 9904 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:40.994359 11101 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
W20260430 07:58:41.082394 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 82455.5us
I20260430 07:58:41.083868 9904 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:40.992007 11050 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.214s user 0.023s sys 0.028s
I20260430 07:58:41.002849 11098 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:41.004333 11084 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:41.027376 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:41.045300 11008 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:41.046370 11137 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
W20260430 07:58:41.050438 9857 consensus_meta.cc:287] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.648s user 0.012s sys 0.000s
I20260430 07:58:41.050900 10998 consensus_queue.cc:237] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.053169 10995 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:41.018730 11092 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:41.054041 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:41.054924 11096 consensus_meta.cc:287] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.538s user 0.006s sys 0.004s
I20260430 07:58:41.055418 11044 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:41.055284 9929 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487"
dest_uuid: "77e19ab653f540339c1c613fad1451b5"
mode: GRACEFUL
new_leader_uuid: "d66db415f0f0402fa431349afbf59888"
from {username='slave'} at 127.0.0.1:52704
I20260430 07:58:41.026132 10919 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:41.023406 11130 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:41.025914 11086 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:41.015673 11018 consensus_queue.cc:260] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:40.989722 11141 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.059062 10563 raft_consensus.cc:2368] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:41.062088 11091 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:41.063676 11122 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:41.016460 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1035) took 1293 ms. Trace:
I20260430 07:58:41.044198 9903 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:41.091131 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1031) took 1373 ms. Trace:
I20260430 07:58:41.028451 9861 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:41.092294 9858 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:41.190793 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 146612us
I20260430 07:58:41.193519 9858 raft_consensus.cc:1275] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260430 07:58:41.095992 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 69722.5us
I20260430 07:58:41.100955 9904 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:41.101114 9860 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:41.192691 9863 rpcz_store.cc:276] 0430 07:58:39.718153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.718300 (+ 147us) service_pool.cc:225] Handling call
0430 07:58:39.769617 (+ 51317us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:39.769903 (+ 286us) raft_consensus.cc:1534] Early marking committed up to index 3
0430 07:58:40.666988 (+897085us) log.cc:844] Serialized 14 byte log entry
0430 07:58:41.090784 (+423796us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:41.090841 (+ 57us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:58:41.090882 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:41.090947 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:41.091074 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":417,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":5292,"thread_start_us":422989,"threads_started":1}
W20260430 07:58:41.088816 11052 consensus_meta.cc:287] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.639s user 0.012s sys 0.000s
I20260430 07:58:41.085845 11057 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:41.088239 11043 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:41.089100 11009 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:40.962425 9856 rpcz_store.cc:276] 0430 07:58:38.764766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:38.766435 (+ 1669us) service_pool.cc:225] Handling call
0430 07:58:38.803415 (+ 36980us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:38.830010 (+ 26595us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:58:40.014974 (+1184964us) log.cc:844] Serialized 14 byte log entry
0430 07:58:40.857454 (+842480us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:40.859673 (+ 2219us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:40.864350 (+ 4677us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:40.881112 (+ 16762us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:40.895232 (+ 14120us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":186438,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":5983,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":4194,"thread_start_us":1862525,"threads_started":2,"wal-append.queue_time_us":888227}
I20260430 07:58:41.205227 9978 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:41.096293 11050 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.320s user 0.024s sys 0.028s
I20260430 07:58:41.092777 9862 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:41.206259 10534 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:58:41.242754 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 141824us
W20260430 07:58:41.275059 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:41.284649 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 43495.2us
I20260430 07:58:41.095534 9854 raft_consensus.cc:1275] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:41.203315 11024 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:41.094794 11002 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:41.147544 10959 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:41.346673 10959 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:41.346853 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:41.347047 11151 log.cc:927] Time spent T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260430 07:58:41.347152 10959 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:41.098757 11101 raft_consensus.cc:697] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:41.206493 9855 rpcz_store.cc:276] 0430 07:58:39.736732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.754611 (+ 17879us) service_pool.cc:225] Handling call
0430 07:58:41.001721 (+1247110us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:41.346261 11149 log.cc:927] Time spent T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.168s user 0.000s sys 0.000s
I20260430 07:58:41.208102 10919 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.286164 11137 raft_consensus.cc:697] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:58:41.308216 11103 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.016s sys 0.015s
I20260430 07:58:41.096698 11023 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:41.188788 9857 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:41.352758 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 146492us
I20260430 07:58:41.345212 9904 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:41.362614 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:41.351735 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 146510us
I20260430 07:58:41.205202 10989 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:41.354126 9903 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:41.369482 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 415683us
I20260430 07:58:41.291621 11018 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.292001 9929 raft_consensus.cc:606] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Received request to transfer leadership to TS d66db415f0f0402fa431349afbf59888
I20260430 07:58:41.170308 10533 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:58:41.357914 11049 consensus_meta.cc:287] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.755s user 0.008s sys 0.004s
I20260430 07:58:41.207847 11008 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.237242 11086 raft_consensus.cc:697] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:41.302736 11092 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:41.357203 11084 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:41.357659 11122 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:41.102113 11114 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.272s
I20260430 07:58:41.207620 11154 consensus_queue.cc:1059] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.167s
I20260430 07:58:41.393445 9978 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:41.372522 11103 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s user 0.019s sys 0.017s
I20260430 07:58:41.371029 11024 raft_consensus.cc:697] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:58:41.405510 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 510384us
W20260430 07:58:41.406040 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 51990.7us
I20260430 07:58:41.377521 11023 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:58:41.412202 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 67058.8us
I20260430 07:58:41.419736 9978 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:41.394893 9857 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1033) took 1671 ms (client timeout 1880 ms). Trace:
W20260430 07:58:41.432974 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:41.395995 11108 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:41.413074 11122 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:41.449236 11122 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.449575 9904 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:41.419534 11101 consensus_queue.cc:237] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.405344 9929 raft_consensus.cc:614] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:58:41.404206 11096 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:41.452373 11096 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.444272 11023 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.432176 10995 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:41.454696 11096 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:41.449230 11108 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.456315 11061 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:41.427974 11057 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:41.457702 11108 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:41.457077 11061 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:41.425848 11111 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:41.461576 11111 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:41.461987 11111 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.462530 11061 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:41.411259 11139 raft_consensus.cc:697] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:41.463747 11139 consensus_queue.cc:237] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.395798 11091 consensus_queue.cc:1048] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260430 07:58:41.395965 10989 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.467768 10005 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260430 07:58:41.470077 10919 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:41.409502 11049 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.474085 9977 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:41.475229 11093 consensus_queue.cc:1048] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260430 07:58:41.476212 11143 log.cc:927] Time spent T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
W20260430 07:58:41.477406 11150 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20260430 07:58:41.477563 11167 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:41.478349 11167 raft_consensus.cc:697] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:41.479557 11167 consensus_queue.cc:237] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.480300 11008 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:41.481427 10995 raft_consensus.cc:697] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:41.482950 10995 consensus_queue.cc:237] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:41.485953 11153 log.cc:927] Time spent T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20260430 07:58:41.486240 11009 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:41.487010 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:41.487428 11141 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:41.445003 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 295196us
I20260430 07:58:41.488813 10533 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:41.491523 10533 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:41.481472 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:41.445785 9857 rpcz_store.cc:269] 0430 07:58:39.720915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.724034 (+ 3119us) service_pool.cc:225] Handling call
0430 07:58:40.293697 (+569663us) spinlock_profiling.cc:243] Waited 5.37 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8a3e 000000000035a3c9 00007fb56e1e08e0 00007fb56e2a9784 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:58:41.392610 (+1098913us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12360064}
W20260430 07:58:41.497148 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 47632.2us
I20260430 07:58:41.394501 11052 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260430 07:58:41.413847 11155 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:41.418406 11138 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:41.501204 11140 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.536690 11140 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:41.503763 11052 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:41.537530 9929 raft_consensus.cc:1275] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:41.537683 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:41.538288 11052 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:41.540833 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:41.505582 11009 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260430 07:58:41.541152 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:41.542922 10567 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:41.506006 11155 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:41.544279 11155 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.546078 11024 consensus_queue.cc:237] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.539098 11052 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:41.538435 9938 raft_consensus.cc:2376] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2. Re-sending same reply.
I20260430 07:58:41.504395 11086 consensus_queue.cc:237] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.506302 11138 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:41.506206 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 86973.8us
I20260430 07:58:41.561985 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:41.506436 10994 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
W20260430 07:58:41.506939 11158 log.cc:927] Time spent T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260430 07:58:41.507678 11049 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:41.507717 10989 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:41.562500 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:41.562693 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:41.563320 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:41.573665 9938 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:41.563436 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:41.520509 11157 log.cc:927] Time spent T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20260430 07:58:41.571748 9932 raft_consensus.cc:2955] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:41.523615 10928 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:41.576315 10928 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:41.576807 10928 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.499601 9904 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:41.511634 11103 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.069s
W20260430 07:58:41.517422 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 124555us
I20260430 07:58:41.584579 9828 leader_election.cc:400] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:41.585026 9828 leader_election.cc:403] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260430 07:58:41.586997 11049 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.591578 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:41.616308 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:41.617017 9935 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:41.618734 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:41.638062 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 138461us
W20260430 07:58:41.669463 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 102798us
I20260430 07:58:41.670743 9978 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:41.679714 9828 leader_election.cc:336] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.696s (SENT)
I20260430 07:58:41.680634 9828 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:41.690857 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:41.692610 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:41.497226 11118 consensus_meta.cc:287] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.621s user 0.004s sys 0.005s
I20260430 07:58:41.696614 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:41.720767 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:41.720853 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:41.721433 10006 raft_consensus.cc:2368] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:41.721565 10003 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
W20260430 07:58:41.711504 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 68736.2us
W20260430 07:58:41.709444 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:41.718894 9935 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:41.720301 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:41.499847 11154 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:41.638068 9937 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:41.737269 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:41.718351 9828 leader_election.cc:336] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.703s (SENT)
I20260430 07:58:41.738839 10003 raft_consensus.cc:1275] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260430 07:58:41.743292 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 72660.9us
I20260430 07:58:41.775344 9978 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:41.776278 9828 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:41.565090 11138 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:41.713325 11084 raft_consensus.cc:697] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:41.593034 11002 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:41.711328 10994 raft_consensus.cc:697] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:41.711639 11006 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:41.724143 11108 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:41.709928 11035 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:41.497699 11137 consensus_queue.cc:237] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:41.825755 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 101170us
W20260430 07:58:41.826349 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 51097.9us
I20260430 07:58:41.996361 11035 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:41.997092 9977 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:41.998608 11035 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:41.999873 11090 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:41.786155 11084 consensus_queue.cc:237] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:42.002604 11090 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.004824 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:42.005540 11139 consensus_queue.cc:1048] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:41.910111 11093 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:42.007808 11093 raft_consensus.cc:697] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:42.001019 11161 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:42.008924 11161 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.009478 11093 consensus_queue.cc:237] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.011484 11161 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:42.005963 10566 raft_consensus.cc:2393] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:41.801668 11141 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
W20260430 07:58:41.975366 11018 consensus_meta.cc:287] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.579s user 0.013s sys 0.000s
I20260430 07:58:41.800123 11096 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260430 07:58:42.049372 10565 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.079s sys 0.243s
W20260430 07:58:42.049880 10565 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.508s user 0.079s sys 0.243s
I20260430 07:58:41.956609 11009 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:41.956266 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:41.911762 11059 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:42.055606 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:41.729774 11118 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:42.065155 9977 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:42.015707 11035 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:41.755430 10919 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.295s
W20260430 07:58:41.980360 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 488643us
I20260430 07:58:42.068740 10006 raft_consensus.cc:2393] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:42.049964 11090 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:42.075341 11167 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260430 07:58:42.057406 11052 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:42.089819 10994 consensus_queue.cc:237] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.075372 11035 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:41.909682 11002 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:42.076092 11059 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.067276 11009 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:42.067312 11141 raft_consensus.cc:697] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:42.101650 11061 raft_consensus.cc:3055] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:42.082595 10998 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260430 07:58:42.081314 10959 consensus_meta.cc:287] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.734s user 0.008s sys 0.004s
W20260430 07:58:42.090883 11111 consensus_meta.cc:287] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.628s user 0.010s sys 0.000s
W20260430 07:58:42.112003 11155 consensus_meta.cc:287] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.567s user 0.005s sys 0.010s
W20260430 07:58:42.128322 11023 consensus_meta.cc:287] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.674s user 0.003s sys 0.006s
I20260430 07:58:42.129894 11141 consensus_queue.cc:237] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:42.094998 11122 consensus_meta.cc:287] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.645s user 0.010s sys 0.002s
I20260430 07:58:42.130951 11059 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:42.113631 11101 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.006s sys 0.019s
I20260430 07:58:42.122260 11061 raft_consensus.cc:740] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:58:42.147719 11101 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s user 0.006s sys 0.019s
I20260430 07:58:42.135247 11111 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:42.157055 11118 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:42.137326 11155 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.143000 11122 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.202433 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260430 07:58:42.204582 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 485778us
W20260430 07:58:42.206132 9828 leader_election.cc:336] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.830s (SENT)
I20260430 07:58:42.206890 9828 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:42.203002 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:42.219180 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:42.221482 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:58:42.222920 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:42.224251 9854 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:42.223359 9761 catalog_manager.cc:5671] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 27f97a330f374849be58a168318c1ec6 (127.8.119.129). New cstate: current_term: 2 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
W20260430 07:58:41.832440 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:42.224830 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:58:42.235445 9761 catalog_manager.cc:5671] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 1 to 2, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
I20260430 07:58:42.244567 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.226318 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:42.228138 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:42.228752 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 505752us
I20260430 07:58:42.249100 9860 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:42.249037 9856 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:42.229061 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:42.228139 9859 raft_consensus.cc:2393] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:42.251366 9861 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:42.258010 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.259316 9935 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:42.260962 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.230703 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:42.231462 9929 raft_consensus.cc:2393] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:42.262971 9862 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
W20260430 07:58:42.265815 9859 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1043) took 1266 ms (client timeout 1613 ms). Trace:
I20260430 07:58:42.266170 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:42.266384 9859 rpcz_store.cc:269] 0430 07:58:40.999499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:40.999665 (+ 166us) service_pool.cc:225] Handling call
0430 07:58:42.265531 (+1265866us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400356954. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:42.265736 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2818940800}
I20260430 07:58:42.267019 9936 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:42.270426 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.273586 9937 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:42.272029 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.277751 9932 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:42.283561 9938 raft_consensus.cc:1275] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:42.287374 9862 raft_consensus.cc:1275] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:42.287946 9860 pending_rounds.cc:85] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 2
I20260430 07:58:42.279369 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:42.293967 9860 pending_rounds.cc:107] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20260430 07:58:42.294498 9860 raft_consensus.cc:2917] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20260430 07:58:42.295444 9860 raft_consensus.cc:2889] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:42.296702 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.298218 9929 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:42.301373 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.302125 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.302404 9934 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:42.304415 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:42.305648 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.361956 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.363169 9938 raft_consensus.cc:2368] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:42.364274 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.365072 9938 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:42.310057 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.098510 10533 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:42.111620 10560 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260430 07:58:42.075526 10561 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:42.105517 10566 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 1 LEARNER]: Advancing to term 2
I20260430 07:58:42.130975 11002 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:42.134689 10959 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:42.133761 10928 consensus_meta.cc:287] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.554s user 0.003s sys 0.010s
I20260430 07:58:42.323869 11008 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:42.310902 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:42.136607 11023 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:42.331780 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:42.330828 11061 consensus_queue.cc:260] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.431645 9938 raft_consensus.cc:2410] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260430 07:58:42.336855 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:42.132608 10998 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:42.370841 9932 raft_consensus.cc:2393] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:42.397207 9937 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:42.431414 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.320884 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:42.432937 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.352216 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:42.441037 9936 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:42.354188 11009 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:42.451097 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.354880 11111 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:42.345244 10919 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:42.459110 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.460044 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.461130 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:42.471025 11057 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:42.471614 11057 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.476672 11122 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:42.479497 9856 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:42.481197 10998 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:42.483832 10011 raft_consensus.cc:2393] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:42.484795 11108 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:42.487583 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.488629 9929 raft_consensus.cc:2368] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:42.350455 10972 consensus_queue.cc:1048] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.131s
I20260430 07:58:42.491735 11155 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:42.516772 10008 raft_consensus.cc:2410] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260430 07:58:42.517560 11035 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:42.520442 11035 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:42.521857 10009 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:42.522987 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.524300 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.525032 9936 raft_consensus.cc:2368] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:42.525467 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.526262 9930 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:42.526361 9759 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53852 (request call id 89) took 4281 ms. Trace:
I20260430 07:58:42.531776 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.533262 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:42.534219 10567 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:42.534369 11114 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:42.492755 10005 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:42.537267 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1296) took 1005 ms. Trace:
I20260430 07:58:42.377022 10928 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:42.376997 11008 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.533511 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.552386 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.555034 9931 raft_consensus.cc:1275] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:42.527797 9759 rpcz_store.cc:276] 0430 07:58:38.244962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:38.246002 (+ 1040us) service_pool.cc:225] Handling call
0430 07:58:42.526101 (+4280099us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:58:38.896957 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:58:38.931726 (+ 34769us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:58:38.931760 (+ 34us) write_op.cc:435] Acquired schema lock
0430 07:58:38.931779 (+ 19us) tablet.cc:662] Decoding operations
0430 07:58:38.938530 (+ 6751us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:58:38.942836 (+ 4306us) write_op.cc:641] Partition lock acquired for write op
0430 07:58:38.976562 (+ 33726us) tablet.cc:685] Acquiring locks for 4 operations
0430 07:58:39.022052 (+ 45490us) tablet.cc:701] Row locks acquired
0430 07:58:39.025838 (+ 3786us) write_op.cc:260] PREPARE: finished
0430 07:58:39.041350 (+ 15512us) write_op.cc:270] Start()
0430 07:58:39.057900 (+ 16550us) write_op.cc:276] Timestamp: P: 1777535919041245 usec, L: 0
0430 07:58:39.057948 (+ 48us) op_driver.cc:348] REPLICATION: starting
0430 07:58:39.060139 (+ 2191us) log.cc:844] Serialized 1685 byte log entry
0430 07:58:39.965116 (+904977us) op_driver.cc:464] REPLICATION: finished
0430 07:58:41.033782 (+1068666us) write_op.cc:301] APPLY: starting
0430 07:58:41.072245 (+ 38463us) tablet.cc:1367] starting BulkCheckPresence
0430 07:58:41.079558 (+ 7313us) tablet.cc:1370] finished BulkCheckPresence
0430 07:58:41.103954 (+ 24396us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:58:41.347579 (+243625us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:58:41.347814 (+ 235us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0430 07:58:41.347901 (+ 87us) write_op.cc:312] APPLY: finished
0430 07:58:41.348183 (+ 282us) log.cc:844] Serialized 40 byte log entry
0430 07:58:41.509868 (+161685us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:41.510012 (+ 144us) write_op.cc:454] Released schema lock
0430 07:58:41.510059 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0430 07:58:41.735122 (+225063us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:41.751595 (+ 16473us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":257603,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":1044439,"apply.run_cpu_time_us":54900,"apply.run_wall_time_us":682856,"num_ops":4,"prepare.queue_time_us":213244,"prepare.run_cpu_time_us":5506,"prepare.run_wall_time_us":191619,"replication_time_us":813528,"spinlock_wait_cycles":209152,"thread_start_us":1190733,"threads_started":3,"wal-append.queue_time_us":161146,"wal-append.run_cpu_time_us":1605,"wal-append.run_wall_time_us":1006368}]]}
I20260430 07:58:42.556380 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1291) took 1114 ms. Trace:
I20260430 07:58:42.562486 9860 raft_consensus.cc:1275] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:42.563395 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:42.523023 10560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54770 (request call id 256) took 1082 ms. Trace:
I20260430 07:58:42.517480 10959 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:42.566699 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.541742 10008 rpcz_store.cc:276] 0430 07:58:41.531519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:41.649513 (+117994us) service_pool.cc:225] Handling call
0430 07:58:42.320292 (+670779us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b58000a2098. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:58:42.537199 (+216907us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70297472}
I20260430 07:58:42.536702 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:42.587368 9933 raft_consensus.cc:2368] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:42.588227 10567 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:42.571408 10011 rpcz_store.cc:276] 0430 07:58:41.441685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:41.442056 (+ 371us) service_pool.cc:225] Handling call
0430 07:58:42.556144 (+1114088us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x7b540003f054. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:42.556300 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2267812736}
I20260430 07:58:42.592621 9930 raft_consensus.cc:2368] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:42.595000 9938 raft_consensus.cc:2376] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2. Re-sending same reply.
I20260430 07:58:42.579164 9935 raft_consensus.cc:2376] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2. Re-sending same reply.
I20260430 07:58:42.618891 9936 raft_consensus.cc:1275] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:42.620453 9934 raft_consensus.cc:2376] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2. Re-sending same reply.
I20260430 07:58:42.689601 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:42.688256 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 481986us
W20260430 07:58:42.675757 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 121266us
I20260430 07:58:42.703274 10563 raft_consensus.cc:1240] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:42.705255 10563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54754 (request call id 167) took 1165 ms. Trace:
I20260430 07:58:42.620464 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:42.651850 10928 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:42.707165 9854 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:42.652938 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:42.640511 10989 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:58:42.634034 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 489016us
I20260430 07:58:42.732613 11122 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:42.649171 10919 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.573392 10560 rpcz_store.cc:276] 0430 07:58:41.440431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:41.493512 (+ 53081us) service_pool.cc:225] Handling call
0430 07:58:41.493869 (+ 357us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:42.111243 (+617374us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b540047f520. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5691fb1a0 00007fb5692855a0 00007fb569286611 00007fb56928316f 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c
0430 07:58:42.371291 (+260048us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:42.522572 (+151281us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b540047ed54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:42.522840 (+ 268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1330968192}
I20260430 07:58:42.762109 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:42.770867 9855 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:42.772275 10566 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 2 LEARNER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260430 07:58:42.775225 11049 consensus_meta.cc:287] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 1.180s user 0.011s sys 0.011s
I20260430 07:58:42.775523 9903 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:42.775904 11049 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:42.776594 9977 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:58:42.766929 9828 leader_election.cc:336] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.837s (SENT)
I20260430 07:58:42.777846 9828 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:42.778129 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 45959.7us
I20260430 07:58:42.747839 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:42.783473 9977 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:42.758029 11155 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:42.792789 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:42.797974 9977 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:42.786652 10959 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:42.715365 10563 rpcz_store.cc:276] 0430 07:58:41.539835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:41.539973 (+ 138us) service_pool.cc:225] Handling call
0430 07:58:41.540343 (+ 370us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:42.703894 (+1163551us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:42.704912 (+ 1018us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001171d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:42.705187 (+ 275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2460465920}
I20260430 07:58:42.769889 10989 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.819734 11023 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:42.817919 11008 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:42.826454 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 56) took 1318 ms. Trace:
I20260430 07:58:42.832247 9857 rpcz_store.cc:276] 0430 07:58:41.508400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:41.523407 (+ 15007us) service_pool.cc:225] Handling call
0430 07:58:41.523855 (+ 448us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:41.524280 (+ 425us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:41.524313 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:42.248537 (+724224us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b2c000014c8. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb567b7dc81 00007fb567b8d5e2 00007fb5692bede1 00007fb56928381a 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22
0430 07:58:42.263497 (+ 14960us) log.cc:844] Serialized 26 byte log entry
0430 07:58:42.825657 (+562160us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b540019e4a0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ed470 00007fb5691faa98 00007fb569283d0b 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac
0430 07:58:42.825732 (+ 75us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:58:42.825787 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:42.825951 (+ 164us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:42.826036 (+ 85us) raft_consensus.cc:1685] finished
0430 07:58:42.826050 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:42.826257 (+ 207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":433,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"spinlock_wait_cycles":622397184,"thread_start_us":167507,"threads_started":1,"wal-append.queue_time_us":159815}
I20260430 07:58:42.796269 11155 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.834870 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:42.863502 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:42.842185 9860 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:42.823918 10989 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:42.818150 11049 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:42.857425 9855 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:42.814081 11198 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:42.860620 11108 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:42.868902 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 49978.4us
I20260430 07:58:42.818511 11035 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:42.830789 11057 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:42.906226 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.909981 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:42.908869 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 173785us
I20260430 07:58:42.871616 11114 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:42.818929 11122 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.960707 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:42.961802 9857 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 3
I20260430 07:58:42.961889 10989 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:42.962826 10989 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:42.967458 9828 leader_election.cc:336] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.847s (SENT)
I20260430 07:58:42.968250 9828 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:42.947978 11198 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:42.973759 11198 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:42.974643 10010 raft_consensus.cc:1275] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:42.958504 11161 raft_consensus.cc:2804] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260430 07:58:42.959895 11205 log.cc:927] Time spent T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20260430 07:58:42.977942 9862 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:42.940996 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.958562 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:42.987857 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.994316 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.994738 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:42.996299 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:42.948576 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:42.997396 10008 raft_consensus.cc:2393] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
W20260430 07:58:42.979998 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 72999us
I20260430 07:58:42.947567 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:43.002590 10919 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:43.010757 10998 consensus_meta.cc:287] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.528s user 0.007s sys 0.005s
I20260430 07:58:43.011277 10998 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:42.983407 11161 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:43.024009 9860 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.024940 10998 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:43.025117 11208 log.cc:927] Time spent T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260430 07:58:43.026292 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:43.027506 10010 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:43.028358 11155 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:43.030145 9855 raft_consensus.cc:1275] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:43.034497 9856 raft_consensus.cc:2393] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:43.036101 9856 raft_consensus.cc:1275] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:42.979563 11035 raft_consensus.cc:697] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:43.038287 11035 consensus_queue.cc:237] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:42.952625 10959 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20260430 07:58:43.042881 9934 raft_consensus.cc:2410] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260430 07:58:43.015334 11108 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:43.044052 11108 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:43.040845 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 73478.1us
I20260430 07:58:43.044864 11108 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.046139 11161 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.078003 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:43.092918 9828 leader_election.cc:336] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.871s (SENT)
I20260430 07:58:43.093710 9828 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:43.101588 9855 raft_consensus.cc:1275] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:43.103334 9855 raft_consensus.cc:1275] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:43.103478 9856 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:43.073166 10998 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:43.103711 10994 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:43.104992 9855 raft_consensus.cc:1275] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:43.080907 10919 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:43.132457 9860 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:43.134107 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 52614.4us
I20260430 07:58:43.135146 9933 raft_consensus.cc:2368] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
W20260430 07:58:43.135387 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 53430.3us
I20260430 07:58:43.143464 10003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1299) took 1309 ms. Trace:
I20260430 07:58:43.097074 9854 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:43.163087 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:43.162875 10003 rpcz_store.cc:276] 0430 07:58:41.788047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:41.788320 (+ 273us) service_pool.cc:225] Handling call
0430 07:58:42.304892 (+516572us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b58000a2098. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:42.305365 (+ 473us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:42.306247 (+ 882us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:42.306282 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:42.307032 (+ 750us) log.cc:844] Serialized 26 byte log entry
0430 07:58:42.569153 (+262121us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:58:42.569206 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:42.569791 (+ 585us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:43.066140 (+496349us) raft_consensus.cc:1685] finished
0430 07:58:43.066173 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:43.097440 (+ 31267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":69996928,"thread_start_us":261651,"threads_started":1,"wal-append.queue_time_us":489928}
I20260430 07:58:43.182269 10934 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:43.119874 11008 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:43.169430 11049 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:43.183030 10934 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.199535 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:43.200286 11101 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:43.202034 9937 raft_consensus.cc:2393] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:58:43.209896 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 144609us
I20260430 07:58:43.211419 9933 raft_consensus.cc:1275] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260430 07:58:43.199407 11210 log.cc:927] Time spent T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.221s user 0.000s sys 0.001s
I20260430 07:58:43.141119 10998 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:43.217914 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 318919us
I20260430 07:58:43.225917 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:43.198127 11108 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:43.255990 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:43.167284 10994 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:43.268013 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:43.179960 10919 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.275746 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.226107 9857 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:43.287679 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:43.289413 9859 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:43.292047 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:43.292627 11018 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:58:43.293208 9934 raft_consensus.cc:2368] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:43.301581 9937 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.254428 11008 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.317378 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:43.320855 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.321105 9903 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:43.310451 9828 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.176s (SENT) [suppressed 25 similar messages]
I20260430 07:58:43.328670 10972 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:43.329037 10534 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:43.313128 11092 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.135s
I20260430 07:58:43.330155 11114 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:43.329152 10972 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:43.330480 9860 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:43.330677 11114 raft_consensus.cc:697] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:43.330826 10972 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:43.331833 11114 consensus_queue.cc:237] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:43.313022 11057 consensus_queue.cc:1048] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
W20260430 07:58:43.333621 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:43.271962 10533 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:43.279510 11101 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:43.338809 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 154870us
W20260430 07:58:43.339730 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:43.340494 11009 raft_consensus.cc:2804] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:43.293923 11163 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:43.344991 9862 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:43.347376 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 164200us
I20260430 07:58:43.318143 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:43.341087 11009 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:43.370337 9977 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:43.376791 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:43.386377 10919 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:43.387764 10919 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:43.388420 10919 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.366658 11009 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:43.390427 10994 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:43.391237 9978 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:43.392288 10998 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:43.392905 10998 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:43.339773 11101 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:43.401877 9938 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:43.394192 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:43.394758 10003 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:43.393294 10998 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:43.293313 11141 consensus_queue.cc:1059] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.191s
I20260430 07:58:43.375384 9932 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:43.412578 9978 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:43.431900 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:43.406216 9854 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.441213 9937 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:43.441222 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 77291us
I20260430 07:58:43.361068 11163 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.413416 10998 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.430327 10934 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:43.445231 9937 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:43.436766 11219 log.cc:927] Time spent T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.260s user 0.001s sys 0.001s
I20260430 07:58:43.440686 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:43.460865 11023 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:43.482161 9929 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:43.487239 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:43.493018 9938 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
W20260430 07:58:43.500658 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 107327us
I20260430 07:58:43.502221 10005 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:43.507360 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:43.485877 11023 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:43.509426 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 213163us
I20260430 07:58:43.513365 11023 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:43.514834 10972 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:43.511391 11213 log.cc:927] Time spent T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.234s user 0.000s sys 0.001s
I20260430 07:58:43.512259 11008 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:43.519034 10009 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.523249 9828 leader_election.cc:400] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:43.524108 9828 leader_election.cc:403] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260430 07:58:43.525293 11108 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.525588 9978 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:43.528298 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:43.499046 10007 raft_consensus.cc:1275] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:43.530539 11023 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.532505 9828 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:43.534346 11023 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:43.535893 11049 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:43.536474 11049 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:43.536787 11049 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.538170 11108 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
W20260430 07:58:43.539633 11198 consensus_meta.cc:287] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.565s user 0.009s sys 0.001s
I20260430 07:58:43.540042 11198 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.541110 9935 raft_consensus.cc:1240] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:43.511559 11212 log.cc:927] Time spent T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.218s user 0.001s sys 0.000s
I20260430 07:58:43.543381 10919 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:43.543072 11008 consensus_queue.cc:1048] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:43.544157 9857 raft_consensus.cc:1275] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:43.545459 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.546341 9857 raft_consensus.cc:2393] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
W20260430 07:58:43.546873 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:43.497634 10994 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:43.535878 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:43.548013 10994 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:43.548456 10994 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.549355 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:43.551834 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:43.552500 9863 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.552282 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:43.553794 11163 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:43.553938 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.507658 11155 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:43.554752 11155 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.499907 11122 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:43.555116 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:43.555662 11205 log.cc:927] Time spent T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260430 07:58:43.557928 11209 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.738s user 0.006s sys 0.000s
I20260430 07:58:43.563792 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:43.555186 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:43.565624 10007 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.564280 11049 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.566089 11226 raft_consensus.cc:3055] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:43.566469 11226 raft_consensus.cc:740] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:43.567318 11226 consensus_queue.cc:260] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:43.567960 11049 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:43.568631 11226 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.569967 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1063) took 1354 ms. Trace:
I20260430 07:58:43.571162 11061 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.571838 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:43.573866 9934 raft_consensus.cc:1240] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:43.570467 9861 rpcz_store.cc:276] 0430 07:58:42.215665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:42.288930 (+ 73265us) service_pool.cc:225] Handling call
0430 07:58:42.291605 (+ 2675us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:42.491203 (+199598us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:42.491240 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:42.491764 (+ 524us) log.cc:844] Serialized 26 byte log entry
0430 07:58:42.744943 (+253179us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:42.745001 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:42.745093 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:43.557228 (+812135us) spinlock_profiling.cc:243] Waited 44 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:58:43.569588 (+ 12360us) raft_consensus.cc:1685] finished
0430 07:58:43.569611 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:43.569906 (+ 295us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":198761,"raft.queue_time_us":235306,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":36916,"spinlock_wait_cycles":101179136,"thread_start_us":252709,"threads_started":1,"wal-append.queue_time_us":327637}
I20260430 07:58:43.575100 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:43.575291 11222 log.cc:927] Time spent T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.208s user 0.000s sys 0.001s
I20260430 07:58:43.575739 9934 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.592049 9863 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:43.595036 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:43.584203 11161 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:43.505381 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:43.601152 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:43.600106 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 88077us
I20260430 07:58:43.600682 9938 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.598968 9937 raft_consensus.cc:1275] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:43.599483 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:43.609790 10005 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:43.586159 10972 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:43.625037 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 89018.4us
I20260430 07:58:43.640962 9977 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:43.617230 11101 consensus_queue.cc:1048] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260430 07:58:43.649464 11009 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:43.652618 10008 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:43.653914 10561 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260430 07:58:43.597366 10998 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.654047 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:43.654415 10561 pending_rounds.cc:85] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Aborting all ops after (but not including) 4
I20260430 07:58:43.654714 10010 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:43.655104 10561 pending_rounds.cc:107] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260430 07:58:43.655622 10561 raft_consensus.cc:2889] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:43.656394 10998 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:43.654521 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:43.664682 9931 raft_consensus.cc:1275] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:43.651329 10007 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:43.629709 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:43.690694 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 73633.8us
W20260430 07:58:43.690404 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 179073us
I20260430 07:58:43.694263 9938 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:43.671305 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:43.696784 9977 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:43.694988 10009 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:43.698915 9933 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:43.703985 9978 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:43.708069 10005 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:43.709378 11050 consensus_queue.cc:1059] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260430 07:58:43.717978 11050 raft_consensus.cc:3055] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:43.713760 9936 raft_consensus.cc:2410] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:58:43.675797 11161 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:43.675781 11220 log.cc:927] Time spent T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.380s user 0.005s sys 0.000s
I20260430 07:58:43.674445 10989 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:43.717675 11006 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260430 07:58:43.723191 10959 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:43.723729 10959 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:43.724252 10959 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:43.718382 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:43.719081 11092 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:43.711148 10006 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:43.714176 11155 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:43.636826 10919 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:43.729184 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:43.737671 9903 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:43.750367 9934 raft_consensus.cc:2410] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260430 07:58:43.749369 9863 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:43.676542 11198 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:43.746151 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 49452.4us
I20260430 07:58:43.758951 9828 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:43.759959 11141 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:43.761112 11236 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:43.718397 11050 raft_consensus.cc:740] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:58:43.776266 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:43.779907 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:43.780951 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:43.781402 11216 log.cc:927] Time spent T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.486s user 0.006s sys 0.007s
I20260430 07:58:43.749970 11092 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.788254 9932 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:43.752482 10919 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:43.790740 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1071) took 1232 ms. Trace:
W20260430 07:58:43.791393 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 87409.8us
I20260430 07:58:43.792166 9862 rpcz_store.cc:276] 0430 07:58:42.558053 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:42.563016 (+ 4963us) service_pool.cc:225] Handling call
0430 07:58:43.790578 (+1227562us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b5400191254. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:43.790681 (+ 103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":932286336}
I20260430 07:58:43.753772 11155 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:43.808454 10994 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:43.809432 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 105510us
I20260430 07:58:43.812700 9854 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:43.794020 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 75670.4us
I20260430 07:58:43.814218 10533 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:43.815188 11236 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:43.816068 10533 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:43.817121 11236 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:43.817952 11236 raft_consensus.cc:697] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:43.771639 11141 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:43.821587 11141 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:43.826232 11242 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:43.826731 11242 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:43.827098 11242 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.828334 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:43.833093 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:43.833803 9862 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.836715 9978 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:43.837704 11122 consensus_queue.cc:1048] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260430 07:58:43.838810 11023 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:43.839346 11050 consensus_queue.cc:260] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.840518 9861 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260430 07:58:43.840988 11050 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:43.841670 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:43.842087 11009 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:43.845139 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.846303 11092 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:43.846570 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.846951 10998 consensus_queue.cc:1048] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:43.847486 9858 raft_consensus.cc:2410] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 3 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260430 07:58:43.848263 10010 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:43.850611 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:43.854434 10003 raft_consensus.cc:1240] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:43.855310 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:43.856480 10005 raft_consensus.cc:1240] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:43.857043 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:43.857813 10565 raft_consensus.cc:2393] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:43.855103 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:43.858453 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:43.794030 10919 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:43.870250 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:43.870291 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.871588 9860 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:43.875991 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:43.883070 10004 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 3
W20260430 07:58:43.884160 9828 leader_election.cc:336] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.931s (SENT)
I20260430 07:58:43.886633 9903 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:43.888630 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.889703 9858 raft_consensus.cc:2393] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:43.890753 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.886656 10007 raft_consensus.cc:1240] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:43.895183 10994 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:43.895519 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.896910 9859 raft_consensus.cc:2393] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:43.903795 9828 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:43.898053 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 67482.3us
I20260430 07:58:43.898253 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:43.867666 11155 raft_consensus.cc:697] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:43.819322 11236 consensus_queue.cc:237] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:43.911968 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.907311 10972 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:43.918395 10972 raft_consensus.cc:697] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:43.918493 11155 consensus_queue.cc:237] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.919515 10972 consensus_queue.cc:237] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.915271 9977 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:43.956127 9862 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:43.910547 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:43.974723 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 106737us
I20260430 07:58:43.972375 9863 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:43.963603 11163 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:43.961836 11108 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:43.985582 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 98976us
I20260430 07:58:43.917214 11141 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.990876 9977 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:43.992899 9860 raft_consensus.cc:1275] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:43.978905 10994 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:43.994424 10011 raft_consensus.cc:1240] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
W20260430 07:58:43.995970 11246 log.cc:927] Time spent T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
W20260430 07:58:43.995970 11228 log.cc:927] Time spent T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.512s user 0.000s sys 0.005s
I20260430 07:58:43.997994 11141 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:43.998570 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:43.983234 11093 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:44.000209 9863 raft_consensus.cc:1275] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:44.007666 11093 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:44.008100 11093 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:44.008424 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.009056 9863 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:58:44.000175 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 115976us
I20260430 07:58:44.000495 9860 raft_consensus.cc:1275] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:44.000501 9978 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:44.006660 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:44.011065 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.011359 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.011868 9863 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:44.012142 9860 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:58:44.013381 9828 leader_election.cc:336] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.932s (SENT)
I20260430 07:58:44.013906 9828 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.014837 9859 raft_consensus.cc:2393] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:43.963943 10959 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:43.991953 9854 raft_consensus.cc:2393] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 2: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:44.026557 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.027276 9858 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:44.027439 9861 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:44.029512 9978 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:44.031926 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.033663 9862 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:44.051369 10009 raft_consensus.cc:1240] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:44.015290 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:44.014357 11092 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:44.017632 10998 raft_consensus.cc:2764] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260430 07:58:44.014360 10994 raft_consensus.cc:697] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:44.052332 11023 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:44.049526 11141 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:44.068329 11141 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:44.068827 11141 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:44.060388 11092 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.070436 11230 consensus_queue.cc:1048] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:44.072113 11092 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:44.054029 11163 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:44.073330 10533 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:44.064086 11250 log.cc:927] Time spent T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
W20260430 07:58:44.065627 11245 log.cc:927] Time spent T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
W20260430 07:58:44.066596 11251 log.cc:927] Time spent T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20260430 07:58:44.067556 11138 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:44.002410 11108 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:44.092440 11108 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:44.076211 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 62846.7us
I20260430 07:58:44.077380 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:44.097077 9858 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:44.081477 10989 consensus_queue.cc:1048] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260430 07:58:44.082047 11249 log.cc:927] Time spent T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20260430 07:58:44.082491 11161 consensus_queue.cc:1048] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:44.085456 9977 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:44.051409 11008 consensus_queue.cc:1059] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260430 07:58:44.109441 10004 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
W20260430 07:58:44.115665 9828 leader_election.cc:336] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.932s (SENT)
I20260430 07:58:44.116343 9828 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.094902 11023 raft_consensus.cc:697] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:44.112011 10998 consensus_queue.cc:1048] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:44.109517 11009 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:44.106740 11101 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260430 07:58:44.111929 10994 consensus_queue.cc:237] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.124497 10989 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:44.129611 11108 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:44.127192 10959 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:44.130793 10972 raft_consensus.cc:3055] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:44.151973 10972 raft_consensus.cc:740] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:44.152045 11227 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260430 07:58:44.159221 11023 consensus_queue.cc:237] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.164644 9856 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:44.156364 10560 raft_consensus.cc:1240] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:44.165815 11225 consensus_queue.cc:1048] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260430 07:58:44.152912 10972 consensus_queue.cc:260] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.133599 11242 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.191280 11242 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:44.192067 10928 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:58:44.169744 11004 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:44.192492 11093 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.193308 11004 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.194227 9932 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:44.194669 9937 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:44.195487 11004 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:44.196693 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:44.175057 11050 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:44.175251 10972 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:44.175936 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:44.227177 9858 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:44.151245 10989 raft_consensus.cc:697] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:44.145012 11239 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
W20260430 07:58:44.180158 9828 leader_election.cc:336] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.981s (SENT)
I20260430 07:58:44.238317 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.185926 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.240216 9854 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:44.180773 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:44.241104 11236 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:58:44.172024 11141 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.204931 9857 raft_consensus.cc:1275] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:44.247291 11050 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:44.197099 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.197510 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:44.257637 9862 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:44.200532 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.259198 9861 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:44.204742 9859 raft_consensus.cc:1275] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:44.206952 10919 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.207803 9860 raft_consensus.cc:1240] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
W20260430 07:58:44.223320 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:44.250406 11050 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:44.269932 10003 raft_consensus.cc:1240] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3. Ops: []
W20260430 07:58:44.275496 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 162220us
W20260430 07:58:44.279639 11266 log.cc:927] Time spent T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.055s user 0.002s sys 0.000s
I20260430 07:58:44.245142 11141 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:44.279971 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 194605us
I20260430 07:58:44.281737 11226 consensus_queue.cc:1048] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:44.282557 9978 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.287281 10011 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:44.287274 9977 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:44.278029 11242 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:44.291477 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:44.291782 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:44.292410 10009 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:44.292469 10560 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:44.295049 10998 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260430 07:58:44.297678 9903 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:44.298691 11008 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:44.299110 11008 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:44.299474 11008 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:44.305717 9858 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:44.308079 10004 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:44.312566 10998 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:44.312655 11141 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:44.313406 10998 raft_consensus.cc:697] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:44.313527 11141 raft_consensus.cc:697] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:44.313982 11122 consensus_queue.cc:1048] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:44.314625 10998 consensus_queue.cc:237] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.314934 11141 consensus_queue.cc:237] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.313774 11271 consensus_queue.cc:1048] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260430 07:58:44.318053 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:44.228492 10989 consensus_queue.cc:237] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.319687 10003 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:44.320851 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:44.321455 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:44.322052 9936 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
W20260430 07:58:44.322423 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 142270us
I20260430 07:58:44.322970 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:44.323117 9828 leader_election.cc:336] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.002s (SENT)
I20260430 07:58:44.323647 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:44.323719 9828 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.318702 10563 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:44.326418 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:44.328151 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:44.332036 9937 raft_consensus.cc:2410] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 3 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260430 07:58:44.335254 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:44.338652 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:44.339363 9938 raft_consensus.cc:2368] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:44.340410 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:44.341477 9929 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:44.341959 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:44.324958 10561 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:44.348461 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:44.348491 10007 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:44.349325 9930 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:44.338683 10934 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260430 07:58:44.333050 9931 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:44.340528 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:44.355047 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:44.347893 9904 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:44.356105 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 45294.4us
I20260430 07:58:44.357086 9978 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.338183 11242 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:44.342598 10566 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:44.359444 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:44.362600 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:44.366037 9760 catalog_manager.cc:5671] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 1 to 2, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:44.373750 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:44.375854 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:44.377473 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:44.378159 9937 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:44.359246 11263 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260430 07:58:44.382985 9761 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48090 (request call id 189) took 6010 ms. Trace:
W20260430 07:58:44.383972 11272 log.cc:927] Time spent T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260430 07:58:44.385572 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:44.386922 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:44.387485 9933 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:44.343448 10533 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:44.390672 11239 raft_consensus.cc:2804] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:44.368511 9760 catalog_manager.cc:5671] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 1 to 2, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
W20260430 07:58:44.390962 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 58322.4us
I20260430 07:58:44.391366 11239 raft_consensus.cc:697] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:44.392828 11093 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:44.392643 11239 consensus_queue.cc:237] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:44.342511 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:44.395529 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 72399.7us
W20260430 07:58:44.412151 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 55083.9us
I20260430 07:58:44.412997 9978 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.414605 10011 raft_consensus.cc:1275] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260430 07:58:44.418803 9828 leader_election.cc:336] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.025s (SENT)
I20260430 07:58:44.420873 10004 raft_consensus.cc:1240] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:44.423803 10011 raft_consensus.cc:1275] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:44.427279 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:44.400970 11057 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:44.428876 9828 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:44.383605 9761 rpcz_store.cc:276] 0430 07:58:38.372769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:38.372935 (+ 166us) service_pool.cc:225] Handling call
0430 07:58:44.382918 (+6009983us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:58:43.060866 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:58:43.133436 (+ 72570us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:58:43.133532 (+ 96us) write_op.cc:435] Acquired schema lock
0430 07:58:43.133567 (+ 35us) tablet.cc:662] Decoding operations
0430 07:58:43.134262 (+ 695us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:58:43.134404 (+ 142us) write_op.cc:641] Partition lock acquired for write op
0430 07:58:43.134466 (+ 62us) tablet.cc:685] Acquiring locks for 2 operations
0430 07:58:43.293578 (+159112us) tablet.cc:701] Row locks acquired
0430 07:58:43.295413 (+ 1835us) write_op.cc:260] PREPARE: finished
0430 07:58:43.297778 (+ 2365us) write_op.cc:270] Start()
0430 07:58:43.322433 (+ 24655us) write_op.cc:276] Timestamp: P: 1777535923295522 usec, L: 0
0430 07:58:43.356839 (+ 34406us) op_driver.cc:348] REPLICATION: starting
0430 07:58:43.467500 (+110661us) log.cc:844] Serialized 928 byte log entry
0430 07:58:44.013377 (+545877us) op_driver.cc:464] REPLICATION: finished
0430 07:58:44.143977 (+130600us) write_op.cc:301] APPLY: starting
0430 07:58:44.144289 (+ 312us) tablet.cc:1367] starting BulkCheckPresence
0430 07:58:44.144989 (+ 700us) tablet.cc:1370] finished BulkCheckPresence
0430 07:58:44.151899 (+ 6910us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:58:44.152627 (+ 728us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:58:44.152867 (+ 240us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 07:58:44.152923 (+ 56us) write_op.cc:312] APPLY: finished
0430 07:58:44.153213 (+ 290us) log.cc:844] Serialized 28 byte log entry
0430 07:58:44.153446 (+ 233us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:44.156456 (+ 3010us) write_op.cc:454] Released schema lock
0430 07:58:44.198934 (+ 42478us) write_op.cc:341] FINISH: Updating metrics
0430 07:58:44.203246 (+ 4312us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:44.203296 (+ 50us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":667644,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":119185,"apply.run_cpu_time_us":5160,"apply.run_wall_time_us":68613,"num_ops":2,"prepare.queue_time_us":438812,"prepare.run_cpu_time_us":16882,"prepare.run_wall_time_us":622529,"raft.queue_time_us":166210,"raft.run_cpu_time_us":29621,"raft.run_wall_time_us":165887,"replication_time_us":625768,"thread_start_us":629562,"threads_started":4,"wal-append.queue_time_us":240597}]]}
I20260430 07:58:44.418680 11035 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:44.434067 11242 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:44.443116 11230 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260430 07:58:44.444033 9828 leader_election.cc:336] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.149s (SENT)
I20260430 07:58:44.444525 9828 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.445488 11242 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260430 07:58:44.445789 9828 leader_election.cc:336] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.159s (SENT)
I20260430 07:58:44.446331 9828 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.427987 10994 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:44.447304 10994 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:44.447746 10994 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:44.448743 11242 raft_consensus.cc:2749] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260430 07:58:44.449874 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 115575us
W20260430 07:58:44.450076 9828 leader_election.cc:336] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.163s (SENT)
W20260430 07:58:44.451067 9828 leader_election.cc:336] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.182s (SENT)
I20260430 07:58:44.451517 9828 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.451947 9904 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:44.453636 11050 consensus_queue.cc:1059] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:44.454650 11050 raft_consensus.cc:3055] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:44.454993 11050 raft_consensus.cc:740] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:44.457183 11242 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260430 07:58:44.458570 9828 leader_election.cc:336] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.209s (SENT)
I20260430 07:58:44.458997 9828 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.455819 11050 consensus_queue.cc:260] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.461041 11050 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:44.459787 11242 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:44.437197 9938 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:44.438593 11043 consensus_queue.cc:1059] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:44.471269 11043 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:44.459098 10919 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:44.463271 9977 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:44.474303 9977 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:44.469738 9933 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:44.479956 9937 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:44.484048 10934 consensus_queue.cc:1059] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:44.484412 9904 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:44.485076 11057 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:44.482282 11270 raft_consensus.cc:1064] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:44.485812 11267 raft_consensus.cc:1064] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032: attempting to promote NON_VOTER d66db415f0f0402fa431349afbf59888 to VOTER
I20260430 07:58:44.485875 11061 consensus_queue.cc:1048] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:44.471872 11043 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.488963 11043 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:44.488564 11270 consensus_queue.cc:237] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:44.494508 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.495399 9863 raft_consensus.cc:2393] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:44.497447 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:44.498795 9828 leader_election.cc:336] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.226s (SENT)
I20260430 07:58:44.500037 9858 raft_consensus.cc:1275] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:58:44.500639 9828 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.502712 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.480997 10972 raft_consensus.cc:3055] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:44.503453 9855 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:44.503784 10972 raft_consensus.cc:740] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:44.504441 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.505107 9860 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:44.504698 10972 consensus_queue.cc:260] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.476758 11271 raft_consensus.cc:2749] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:44.483176 11093 raft_consensus.cc:2804] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:44.516067 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:44.516570 9828 leader_election.cc:336] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.241s (SENT)
I20260430 07:58:44.497051 9931 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:44.519913 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.520613 9860 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:44.511523 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:44.517050 9828 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.525893 9934 raft_consensus.cc:2393] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:44.473712 11008 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:44.567627 9977 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:44.567384 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:44.566617 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:44.505793 11242 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:44.572526 9861 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:44.574806 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 56819.2us
W20260430 07:58:44.578835 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:44.508296 10934 consensus_queue.cc:1243] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer 48e81934d52a4e93be5171ceea193032 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:58:44.583210 9977 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:44.586994 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:44.515695 11093 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:44.604473 9904 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:44.608191 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:44.609095 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 93218.7us
I20260430 07:58:44.610657 10009 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:44.609100 9859 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:44.580271 10972 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:44.582402 10919 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:44.582934 11108 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:44.508363 11057 consensus_queue.cc:1059] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.090s
I20260430 07:58:44.532246 11004 consensus_queue.cc:1048] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20260430 07:58:44.602120 11141 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:44.610148 11093 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:44.646308 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 129702us
W20260430 07:58:44.649309 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:44.674942 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:44.675597 10004 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:44.639917 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:44.684115 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:44.685972 9903 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:44.687659 11093 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:44.663388 9828 connection.cc:254] RPC call timeout handler was delayed by -2.0380251090000456s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
W20260430 07:58:44.689899 9828 connection.cc:254] RPC call timeout handler was delayed by -2.0011251090000428s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
W20260430 07:58:44.690609 9828 connection.cc:254] RPC call timeout handler was delayed by -1.801325109000004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
W20260430 07:58:44.691079 9828 connection.cc:254] RPC call timeout handler was delayed by -1.5727251090000323s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
W20260430 07:58:44.691370 9828 connection.cc:254] RPC call timeout handler was delayed by -1.4269251090000239s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
W20260430 07:58:44.691659 9828 connection.cc:254] RPC call timeout handler was delayed by -1.1910389880000025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19490000000000002s for a response
I20260430 07:58:44.694379 11108 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:44.696018 11108 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:44.680130 11141 raft_consensus.cc:697] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:44.698540 11141 consensus_queue.cc:237] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.703161 11057 consensus_queue.cc:1243] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer 48e81934d52a4e93be5171ceea193032 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260430 07:58:44.641844 9929 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:44.718003 9760 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48094 (request call id 211) took 5572 ms. Trace:
I20260430 07:58:44.718693 10567 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 2 LEARNER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260430 07:58:44.671664 10007 raft_consensus.cc:1275] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260430 07:58:44.651913 10919 raft_consensus.cc:697] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:44.703819 11057 raft_consensus.cc:3055] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:44.727342 11271 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:44.750264 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:44.729043 11093 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:44.730129 9857 raft_consensus.cc:3055] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:44.730279 11108 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:44.730692 9828 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:44.732272 11275 log.cc:927] Time spent T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.132s user 0.002s sys 0.002s
I20260430 07:58:44.732156 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
W20260430 07:58:44.783739 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 193310us
I20260430 07:58:44.734030 11043 raft_consensus.cc:3055] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
W20260430 07:58:44.749253 11276 log.cc:927] Time spent T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.276s user 0.001s sys 0.000s
I20260430 07:58:44.718451 9760 rpcz_store.cc:276] 0430 07:58:39.145847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:39.146385 (+ 538us) service_pool.cc:225] Handling call
0430 07:58:44.717917 (+5571532us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:58:44.452362 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:58:44.452921 (+ 559us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:58:44.452960 (+ 39us) write_op.cc:435] Acquired schema lock
0430 07:58:44.452980 (+ 20us) tablet.cc:662] Decoding operations
0430 07:58:44.453452 (+ 472us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:58:44.453534 (+ 82us) write_op.cc:641] Partition lock acquired for write op
0430 07:58:44.453560 (+ 26us) tablet.cc:685] Acquiring locks for 2 operations
0430 07:58:44.453914 (+ 354us) tablet.cc:701] Row locks acquired
0430 07:58:44.453932 (+ 18us) write_op.cc:260] PREPARE: finished
0430 07:58:44.454073 (+ 141us) write_op.cc:270] Start()
0430 07:58:44.454199 (+ 126us) write_op.cc:276] Timestamp: P: 1777535924454031 usec, L: 0
0430 07:58:44.454225 (+ 26us) op_driver.cc:348] REPLICATION: starting
0430 07:58:44.454918 (+ 693us) log.cc:844] Serialized 928 byte log entry
0430 07:58:44.460932 (+ 6014us) op_driver.cc:464] REPLICATION: finished
0430 07:58:44.470917 (+ 9985us) write_op.cc:301] APPLY: starting
0430 07:58:44.471015 (+ 98us) tablet.cc:1367] starting BulkCheckPresence
0430 07:58:44.471128 (+ 113us) tablet.cc:1370] finished BulkCheckPresence
0430 07:58:44.471306 (+ 178us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:58:44.471729 (+ 423us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:58:44.471916 (+ 187us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0430 07:58:44.471958 (+ 42us) write_op.cc:312] APPLY: finished
0430 07:58:44.483756 (+ 11798us) log.cc:844] Serialized 28 byte log entry
0430 07:58:44.483955 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:44.484051 (+ 96us) write_op.cc:454] Released schema lock
0430 07:58:44.484088 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0430 07:58:44.484471 (+ 383us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:44.484500 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":52810,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":1838,"apply.run_cpu_time_us":2306,"apply.run_wall_time_us":13507,"num_ops":2,"prepare.queue_time_us":26256,"prepare.run_cpu_time_us":2891,"prepare.run_wall_time_us":10744,"replication_time_us":6568,"thread_start_us":16761,"threads_started":1}]]}
I20260430 07:58:44.728345 11006 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260430 07:58:44.780469 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 44367.3us
W20260430 07:58:44.725821 11277 log.cc:927] Time spent T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.312s user 0.001s sys 0.000s
I20260430 07:58:44.784680 9863 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:44.673831 10994 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:44.719828 9931 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:44.784561 9904 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:44.836050 9977 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:44.752287 11057 raft_consensus.cc:740] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:44.750897 10934 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:58:44.954888 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 118877us
I20260430 07:58:44.794063 9856 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:44.792057 11043 raft_consensus.cc:740] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:44.781620 9857 raft_consensus.cc:740] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:58:44.967590 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 236929us
I20260430 07:58:44.978264 10009 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:44.837972 11108 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:44.973568 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:44.833343 11006 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:58:44.957502 10566 consensus_meta.cc:287] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.578s user 0.010s sys 0.001s
W20260430 07:58:44.968412 9902 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.119.190:40689, user_credentials={real_user=slave}} blocked reactor thread for 44032us
W20260430 07:58:44.987295 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:44.980731 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:44.989421 10010 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:44.983000 9863 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:44.999541 11286 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260430 07:58:44.984880 11273 log.cc:927] Time spent T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.487s user 0.001s sys 0.001s
I20260430 07:58:44.951042 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:44.970037 10998 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:44.980374 11008 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:44.983479 11004 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:45.038061 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:45.038882 9935 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:45.027396 9828 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:44.993132 11006 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:44.993633 10566 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:45.008736 10560 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.017s sys 0.015s
I20260430 07:58:45.063179 10010 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:45.064463 9863 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:58:45.036659 11227 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.009s sys 0.026s
I20260430 07:58:45.047880 11108 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:45.036891 10998 raft_consensus.cc:697] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:45.076956 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:45.078372 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:58:45.093827 9863 raft_consensus.cc:2368] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3.
W20260430 07:58:45.079797 9936 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.015s sys 0.026s
W20260430 07:58:45.061482 10560 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s user 0.017s sys 0.016s
I20260430 07:58:45.021536 11286 consensus_queue.cc:1048] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
W20260430 07:58:45.097352 9936 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s user 0.016s sys 0.026s
W20260430 07:58:45.097465 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:45.126538 9977 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:58:45.078495 11227 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s user 0.009s sys 0.026s
I20260430 07:58:45.128710 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:44.979964 10934 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:45.141415 9861 consensus_meta.cc:287] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.568s user 0.010s sys 0.003s
I20260430 07:58:45.146916 9861 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:45.149101 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 135336us
W20260430 07:58:45.168674 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:45.079846 11242 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:45.127575 11061 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:45.052942 10919 consensus_queue.cc:237] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:45.146152 11138 raft_consensus.cc:2804] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
W20260430 07:58:45.171494 11246 log.cc:927] Time spent T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20260430 07:58:45.069015 11004 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:45.203312 11155 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:45.079325 11057 consensus_queue.cc:260] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:45.246850 11050 consensus_meta.cc:287] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.772s user 0.013s sys 0.001s
W20260430 07:58:45.248806 11280 log.cc:927] Time spent T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.463s user 0.003s sys 0.001s
I20260430 07:58:45.097551 9857 consensus_queue.cc:260] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:45.275964 9859 consensus_meta.cc:287] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.661s user 0.010s sys 0.002s
I20260430 07:58:45.235929 11138 raft_consensus.cc:697] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:45.237900 10994 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:45.331259 9859 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:45.211138 11242 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:45.345438 9938 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s user 0.007s sys 0.042s
W20260430 07:58:45.345860 9938 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s user 0.007s sys 0.042s
I20260430 07:58:45.347986 9828 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:45.359421 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 407849us
W20260430 07:58:45.364060 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:45.364763 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:45.365527 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 117222us
I20260430 07:58:45.367833 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:45.369038 9859 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:45.354494 11057 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:45.394006 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 609180us
W20260430 07:58:45.406561 11284 log.cc:927] Time spent T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.447s user 0.003s sys 0.000s
W20260430 07:58:45.409957 11296 log.cc:927] Time spent T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.227s user 0.001s sys 0.000s
W20260430 07:58:45.410785 11294 log.cc:927] Time spent T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.204s user 0.001s sys 0.000s
W20260430 07:58:45.412387 11293 log.cc:927] Time spent T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260430 07:58:45.361229 9857 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:45.416038 10934 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:45.361536 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:45.250725 10998 consensus_queue.cc:237] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:45.363895 11108 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:45.420877 11289 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.171s user 0.002s sys 0.000s
I20260430 07:58:45.367631 11298 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:45.468125 9903 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:45.423614 9828 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:58:45.424706 11290 log.cc:927] Time spent T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260430 07:58:45.427544 9935 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:45.427690 11043 consensus_queue.cc:260] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:45.398319 11138 consensus_queue.cc:237] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:45.431965 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 53182us
I20260430 07:58:45.440007 9978 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:45.425141 9856 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:45.433387 10972 consensus_meta.cc:287] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.795s user 0.008s sys 0.004s
I20260430 07:58:45.442865 9904 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:45.493462 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 53514us
I20260430 07:58:45.494469 9978 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:45.496533 9978 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:45.455261 9859 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:45.448007 11141 consensus_queue.cc:1048] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.008s
I20260430 07:58:45.456353 11292 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:45.456761 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:45.457458 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:45.458851 11242 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:45.484572 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:45.462875 11298 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:45.503166 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:45.521703 9862 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:45.507548 9978 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:45.440505 9931 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:45.483255 11198 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
W20260430 07:58:45.522779 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 84498.8us
I20260430 07:58:45.501296 10560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54754 (request call id 186) took 1040 ms. Trace:
I20260430 07:58:45.533406 9904 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:45.501744 9929 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:45.508330 11004 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:45.514837 11230 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:45.520469 11093 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:45.528985 11043 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:45.520179 9855 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:45.511725 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1099) took 1151 ms. Trace:
I20260430 07:58:45.508553 11108 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:45.520560 10566 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:45.535741 11301 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:45.517313 11226 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:45.602396 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1116) took 1192 ms. Trace:
I20260430 07:58:45.568033 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:45.616147 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 89909.9us
W20260430 07:58:45.677417 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:45.693542 9862 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:45.546254 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1107) took 1193 ms. Trace:
I20260430 07:58:45.640475 11043 raft_consensus.cc:2764] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260430 07:58:45.647640 11298 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:45.656808 11230 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:45.539503 11198 raft_consensus.cc:697] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:45.673739 9935 rpcz_store.cc:276] 0430 07:58:44.410046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.524857 (+1114811us) service_pool.cc:225] Handling call
0430 07:58:45.601728 (+ 76871us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:45.601958 (+ 230us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.602000 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:45.602067 (+ 67us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:58:45.602106 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.602174 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:45.602320 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:45.696213 9937 rpcz_store.cc:276] 0430 07:58:44.352746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.481413 (+128667us) service_pool.cc:225] Handling call
0430 07:58:44.481742 (+ 329us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:44.481945 (+ 203us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:44.481968 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:44.499368 (+ 17400us) log.cc:844] Serialized 26 byte log entry
0430 07:58:44.616260 (+116892us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:44.616293 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:44.616359 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:45.406200 (+789841us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x7b24000664e8. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb566037e94 00007fb566035f9b 00007fb5660c4d39 00007fb5660c49c5 00007fb5660c400d 00007fb568e064ae 00007fb568e1dc2c 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1
0430 07:58:45.533297 (+127097us) raft_consensus.cc:1685] finished
0430 07:58:45.533323 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:45.546160 (+ 12837us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6292864,"thread_start_us":83117,"threads_started":1,"wal-append.queue_time_us":436039}
I20260430 07:58:45.603747 11242 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:45.603519 11093 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:45.667654 11006 consensus_meta.cc:287] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.600s user 0.007s sys 0.003s
W20260430 07:58:45.728848 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 51803.8us
I20260430 07:58:45.514087 11292 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:45.739014 9904 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:45.573624 11299 consensus_queue.cc:1059] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.230s
I20260430 07:58:45.699047 11226 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:45.726822 11093 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:45.756572 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1111) took 1348 ms. Trace:
W20260430 07:58:45.746971 11304 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20260430 07:58:45.735826 11043 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:45.776209 9904 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:45.760157 9857 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260430 07:58:45.755880 11226 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:45.727974 11242 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:45.731447 11006 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:45.759824 11285 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:58:45.874259 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 98112.3us
I20260430 07:58:45.763548 11239 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
W20260430 07:58:45.879401 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:45.906412 9904 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:45.871001 9857 pending_rounds.cc:85] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 4
W20260430 07:58:45.910269 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 402606us
I20260430 07:58:45.916507 9978 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:45.919744 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:45.944692 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:45.945508 9978 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:45.875100 11286 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:45.927642 11285 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:45.919461 9857 pending_rounds.cc:107] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260430 07:58:45.872678 11299 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:58:45.919555 9854 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.035s sys 0.047s
W20260430 07:58:45.930936 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 414470us
W20260430 07:58:45.932282 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:45.943523 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 519730us
I20260430 07:58:45.870231 11043 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:45.958837 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:45.931479 10998 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:45.539207 10560 rpcz_store.cc:276] 0430 07:58:44.460889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.461053 (+ 164us) service_pool.cc:225] Handling call
0430 07:58:44.461448 (+ 395us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:44.461885 (+ 437us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:44.461922 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:44.462423 (+ 501us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.233576 (+771153us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:58:45.238086 (+ 4510us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.249932 (+ 11846us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:45.466753 (+216821us) raft_consensus.cc:1685] finished
0430 07:58:45.476527 (+ 9774us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:45.492272 (+ 15745us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1113,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":4445,"thread_start_us":634354,"threads_started":1,"wal-append.queue_time_us":772647}
I20260430 07:58:45.896368 11198 consensus_queue.cc:237] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:45.959842 11057 consensus_meta.cc:287] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.549s user 0.011s sys 0.000s
I20260430 07:58:45.979468 9978 leader_election.cc:400] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Vote denied by peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679) with higher term. Message: Invalid argument: T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3.
I20260430 07:58:45.953965 11035 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:45.994428 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:45.976255 9854 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s user 0.036s sys 0.048s
I20260430 07:58:45.963418 9857 raft_consensus.cc:2889] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:45.997203 9978 leader_election.cc:403] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260430 07:58:46.000273 9977 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:46.002056 9935 raft_consensus.cc:1275] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260430 07:58:46.002480 9935 pending_rounds.cc:85] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 1
I20260430 07:58:46.002682 9935 pending_rounds.cc:107] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260430 07:58:46.002899 9935 raft_consensus.cc:2889] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:45.976707 10989 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:46.011404 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:46.013768 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:46.014886 10010 raft_consensus.cc:2393] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
W20260430 07:58:45.992388 11309 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
W20260430 07:58:46.016860 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 111752us
I20260430 07:58:45.982832 11239 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:45.619065 9930 rpcz_store.cc:276] 0430 07:58:44.350001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.394618 (+ 44617us) service_pool.cc:225] Handling call
0430 07:58:44.452565 (+ 57947us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:44.471010 (+ 18445us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:44.471045 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:44.471667 (+ 622us) log.cc:844] Serialized 26 byte log entry
0430 07:58:44.840497 (+368830us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:44.923612 (+ 83115us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:44.960578 (+ 36966us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:45.466825 (+506247us) raft_consensus.cc:1685] finished
0430 07:58:45.476584 (+ 9759us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:45.492225 (+ 15641us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25820,"raft.queue_time_us":19271,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":3771,"thread_start_us":362723,"threads_started":1,"wal-append.queue_time_us":606696}
I20260430 07:58:46.028291 9904 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:58:45.989750 11108 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:45.999574 11018 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:45.965471 10919 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260430 07:58:45.964092 11286 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.035708 9828 leader_election.cc:400] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:46.075171 9828 leader_election.cc:403] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260430 07:58:45.999375 11292 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.037832 9903 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:58:46.025264 11305 log.cc:927] Time spent T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.255s user 0.000s sys 0.001s
W20260430 07:58:46.039184 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 74229.6us
I20260430 07:58:46.101157 9978 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:45.993211 10998 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.032313 11035 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:46.103976 10006 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
W20260430 07:58:46.120857 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 81975us
W20260430 07:58:46.121562 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 93309.6us
I20260430 07:58:46.076701 11057 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:46.076514 11310 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:46.077517 11006 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.102896 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 76) took 1377 ms. Trace:
I20260430 07:58:46.102326 11292 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:46.102037 11299 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:46.035406 11239 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.104975 11242 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.146292 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1108) took 1793 ms. Trace:
I20260430 07:58:46.151366 9934 rpcz_store.cc:276] 0430 07:58:44.353016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.526671 (+173655us) service_pool.cc:225] Handling call
0430 07:58:44.590604 (+ 63933us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.036046 (+445442us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.036078 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.075962 (+ 39884us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.452857 (+376895us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.452891 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.452964 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.146069 (+693105us) raft_consensus.cc:1685] finished
0430 07:58:46.146091 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.146231 (+ 140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6509,"raft.queue_time_us":29779,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":9595,"thread_start_us":688256,"threads_started":2,"wal-append.queue_time_us":589007}
I20260430 07:58:46.147886 10989 raft_consensus.cc:697] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:46.120680 11122 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
W20260430 07:58:46.149992 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 123651us
I20260430 07:58:46.161517 9904 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:46.099351 11108 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.118043 11035 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.106398 11018 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.171839 9904 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:46.146615 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:46.174942 9904 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:46.175802 11318 log.cc:927] Time spent T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.171s user 0.000s sys 0.001s
W20260430 07:58:46.176110 11315 log.cc:927] Time spent T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.210s user 0.001s sys 0.000s
I20260430 07:58:46.177489 10972 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:46.177963 10972 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:46.178359 10972 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:46.142191 11310 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.179162 10004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1353) took 1451 ms. Trace:
I20260430 07:58:46.179538 10004 rpcz_store.cc:276] 0430 07:58:44.727669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.732587 (+ 4918us) service_pool.cc:225] Handling call
0430 07:58:44.732973 (+ 386us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:44.751126 (+ 18153us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:44.751156 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.429105 (+677949us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b1800361600. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661b3be6 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb567b7dc81 00007fb567b8d5e2 00007fb5692bede1 00007fb56928381a
0430 07:58:45.429603 (+ 498us) spinlock_profiling.cc:243] Waited 49.1 ms on lock 0x7b2c0008b928. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb567b7dc81 00007fb567b8d5e2 00007fb5692bede1 00007fb56928381a 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22
0430 07:58:45.435284 (+ 5681us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.910810 (+475526us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.910865 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.910936 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.178953 (+268017us) raft_consensus.cc:1685] finished
0430 07:58:46.178974 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.179110 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2558,"raft.queue_time_us":18136,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":13280,"spinlock_wait_cycles":192982272,"thread_start_us":473402,"threads_started":1,"wal-append.queue_time_us":524688}
I20260430 07:58:46.156339 11329 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:46.188114 11299 consensus_queue.cc:1048] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260430 07:58:46.188740 11329 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:46.190478 11316 log.cc:927] Time spent T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260430 07:58:46.191706 11329 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:46.192804 11320 log.cc:927] Time spent T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260430 07:58:46.193218 9934 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02a63423ae464eb7b415a38419524150"
dest_uuid: "77e19ab653f540339c1c613fad1451b5"
mode: GRACEFUL
new_leader_uuid: "48e81934d52a4e93be5171ceea193032"
from {username='slave'} at 127.0.0.1:52704
I20260430 07:58:46.194244 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1111) took 1206 ms. Trace:
I20260430 07:58:46.181098 11310 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.172749 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1107) took 1373 ms. Trace:
I20260430 07:58:46.195223 11043 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:46.195480 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1113) took 1786 ms. Trace:
I20260430 07:58:46.195845 9933 rpcz_store.cc:276] 0430 07:58:44.408850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.202584 (+793734us) service_pool.cc:225] Handling call
0430 07:58:45.203073 (+ 489us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.203423 (+ 350us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.203454 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.353400 (+149946us) spinlock_profiling.cc:243] Waited 6.88 ms on lock 0x7b2c00010988. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb567b7dc81 00007fb567b8d5e2 00007fb5692bede1 00007fb56928381a 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22
0430 07:58:45.362384 (+ 8984us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.864448 (+502064us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.864513 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.864607 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.195155 (+330548us) raft_consensus.cc:1685] finished
0430 07:58:46.195185 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.195423 (+ 238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7860,"raft.queue_time_us":10423,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":17135,"spinlock_wait_cycles":15827328,"thread_start_us":367992,"threads_started":1,"wal-append.queue_time_us":564473}
I20260430 07:58:46.197340 11018 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:46.173738 11006 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:46.198254 11006 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:46.198583 9933 raft_consensus.cc:1275] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:58:46.198886 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1118) took 1632 ms. Trace:
I20260430 07:58:46.198623 11006 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:58:46.200299 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:46.202165 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1117) took 1635 ms. Trace:
I20260430 07:58:46.202571 9931 rpcz_store.cc:276] 0430 07:58:44.566458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.606390 (+1039932us) service_pool.cc:225] Handling call
0430 07:58:45.663178 (+ 56788us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.663652 (+ 474us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.663708 (+ 56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.664344 (+ 636us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.932968 (+268624us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.941547 (+ 8579us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.952406 (+ 10859us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.201905 (+249499us) raft_consensus.cc:1685] finished
0430 07:58:46.201932 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.202085 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8128,"raft.queue_time_us":3917,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":6013,"thread_start_us":260791,"threads_started":1,"wal-append.queue_time_us":409804}
W20260430 07:58:46.204994 11319 log.cc:927] Time spent T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260430 07:58:46.199312 9937 rpcz_store.cc:276] 0430 07:58:44.566773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.702737 (+1135964us) service_pool.cc:225] Handling call
0430 07:58:45.734447 (+ 31710us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.753844 (+ 19397us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.753890 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.774010 (+ 20120us) log.cc:844] Serialized 26 byte log entry
0430 07:58:46.005740 (+231730us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:46.005779 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.005846 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.198686 (+192840us) raft_consensus.cc:1685] finished
0430 07:58:46.198706 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.198833 (+ 127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18274,"raft.queue_time_us":18765,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":6496,"spinlock_wait_cycles":2816,"thread_start_us":102544,"threads_started":1,"wal-append.queue_time_us":310716}
W20260430 07:58:46.207016 11295 log.cc:927] Time spent T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.932s user 0.004s sys 0.000s
I20260430 07:58:46.207587 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:46.208976 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1122) took 1586 ms. Trace:
I20260430 07:58:46.210645 11239 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.209950 9933 rpcz_store.cc:276] 0430 07:58:44.622858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.197763 (+1574905us) service_pool.cc:225] Handling call
0430 07:58:46.198139 (+ 376us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.199091 (+ 952us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.208895 (+ 9804us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:46.211297 11312 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.232s user 0.001s sys 0.001s
W20260430 07:58:46.212661 11314 log.cc:927] Time spent T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.178s user 0.001s sys 0.000s
I20260430 07:58:46.214944 10998 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:46.215214 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1110) took 1807 ms. Trace:
W20260430 07:58:46.215998 11313 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260430 07:58:46.217101 11108 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.157804 11292 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.217906 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1104) took 1571 ms. Trace:
W20260430 07:58:46.220367 11327 log.cc:927] Time spent T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20260430 07:58:46.218726 9858 rpcz_store.cc:276] 0430 07:58:44.646468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.646592 (+ 124us) service_pool.cc:225] Handling call
0430 07:58:44.646930 (+ 338us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:44.652272 (+ 5342us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:44.652300 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:44.667977 (+ 15677us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.009776 (+341799us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.009829 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.197155 (+1187326us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b540010c054. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:46.197202 (+ 47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.197261 (+ 59us) raft_consensus.cc:1685] finished
0430 07:58:46.217609 (+ 20348us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.217849 (+ 240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2261,"raft.queue_time_us":2818,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":5397,"spinlock_wait_cycles":6136576,"thread_start_us":341367,"threads_started":1,"wal-append.queue_time_us":512083}
I20260430 07:58:46.221720 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1114) took 1189 ms. Trace:
I20260430 07:58:46.222618 9828 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:46.186018 11317 log.cc:927] Time spent T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.191s user 0.000s sys 0.001s
W20260430 07:58:46.186868 11325 log.cc:927] Time spent T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
W20260430 07:58:46.187839 11326 log.cc:927] Time spent T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260430 07:58:46.184962 11122 raft_consensus.cc:697] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:46.222173 9860 rpcz_store.cc:276] 0430 07:58:45.032535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.646463 (+613928us) service_pool.cc:225] Handling call
0430 07:58:45.646844 (+ 381us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.647197 (+ 353us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.647228 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.647744 (+ 516us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.931906 (+284162us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.931944 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.932004 (+ 60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.221430 (+289426us) raft_consensus.cc:1685] finished
0430 07:58:46.221461 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.221645 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18709,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":286,"thread_start_us":283542,"threads_started":1,"wal-append.queue_time_us":365156}
W20260430 07:58:46.236840 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 135728us
I20260430 07:58:46.194651 9861 rpcz_store.cc:276] 0430 07:58:44.988356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.365605 (+377249us) service_pool.cc:225] Handling call
0430 07:58:45.365954 (+ 349us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.366282 (+ 328us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.366320 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.393639 (+ 27319us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.896276 (+502637us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.896336 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.896438 (+ 102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.194010 (+297572us) raft_consensus.cc:1685] finished
0430 07:58:46.194038 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.194175 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5419,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":35595,"thread_start_us":430266,"threads_started":1,"wal-append.queue_time_us":569060}
I20260430 07:58:46.218014 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1112) took 1205 ms. Trace:
I20260430 07:58:46.238080 9859 rpcz_store.cc:276] 0430 07:58:45.012199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.497938 (+485739us) service_pool.cc:225] Handling call
0430 07:58:45.498300 (+ 362us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.961982 (+463682us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.962047 (+ 65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.964214 (+ 2167us) log.cc:844] Serialized 26 byte log entry
0430 07:58:46.034111 (+ 69897us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:46.034154 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.034228 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.217740 (+183512us) raft_consensus.cc:1685] finished
0430 07:58:46.217773 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.217942 (+ 169us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18636,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":28164,"thread_start_us":504415,"threads_started":2,"wal-append.queue_time_us":134649}
I20260430 07:58:46.241386 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1119) took 1111 ms. Trace:
W20260430 07:58:46.243705 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 68811us
I20260430 07:58:46.226751 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:46.244737 9904 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:46.212908 9933 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260430 07:58:46.157960 11057 raft_consensus.cc:697] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:46.163116 11242 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:46.159658 11335 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:46.141911 9857 rpcz_store.cc:276] 0430 07:58:44.721914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.724044 (+ 2130us) service_pool.cc:225] Handling call
0430 07:58:44.729729 (+ 5685us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.032155 (+1302426us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.099602 (+ 67447us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:46.248678 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 121079us
I20260430 07:58:46.256259 9759 catalog_manager.cc:5671] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 reported cstate change: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:46.167115 11050 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
I20260430 07:58:46.271510 9934 raft_consensus.cc:606] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Received request to transfer leadership to TS 48e81934d52a4e93be5171ceea193032
I20260430 07:58:46.252305 9858 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:46.252635 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1123) took 1524 ms. Trace:
I20260430 07:58:46.241922 9859 rpcz_store.cc:276] 0430 07:58:45.130453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.240274 (+1109821us) service_pool.cc:225] Handling call
0430 07:58:46.240694 (+ 420us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.240888 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:46.240918 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:46.240981 (+ 63us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:46.241043 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.241130 (+ 87us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.241310 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.273787 9933 rpcz_store.cc:276] 0430 07:58:44.721626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.211986 (+1490360us) service_pool.cc:225] Handling call
0430 07:58:46.212455 (+ 469us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.245913 (+ 33458us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.246212 (+ 299us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.276449 10534 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:46.258438 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:46.277881 9860 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:46.267575 9759 catalog_manager.cc:5671] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 2 to 3, leader changed from d66db415f0f0402fa431349afbf59888 (127.8.119.132) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 3 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:46.252069 11155 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:46.260737 9904 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:46.247290 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:46.281502 9759 catalog_manager.cc:5671] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 1 to 3, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 3 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:46.251286 11108 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:46.251665 11337 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:46.293602 9759 catalog_manager.cc:5671] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 1 to 3, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 3 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:58:46.262208 9978 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:46.262115 11332 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:46.295886 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1115) took 1886 ms. Trace:
I20260430 07:58:46.298069 9759 catalog_manager.cc:5671] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 1 to 3, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:46.270457 11292 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.301712 9938 rpcz_store.cc:276] 0430 07:58:44.409495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.346811 (+937316us) service_pool.cc:225] Handling call
0430 07:58:45.351704 (+ 4893us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.362759 (+ 11055us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.362791 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.366205 (+ 3414us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.765377 (+399172us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.765422 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.765531 (+ 109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.295669 (+530138us) raft_consensus.cc:1685] finished
0430 07:58:46.295692 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.295821 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38392,"raft.queue_time_us":144,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":11956,"thread_start_us":335660,"threads_started":1,"wal-append.queue_time_us":501685}
I20260430 07:58:46.304909 9938 raft_consensus.cc:1275] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:46.308079 9828 leader_election.cc:400] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:46.289234 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:46.308478 9904 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:46.291672 11006 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.284727 11155 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:46.324872 9933 raft_consensus.cc:1275] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:46.295943 11198 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:46.196041 10989 consensus_queue.cc:237] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.326611 9904 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:46.329851 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 67844.3us
I20260430 07:58:46.332057 9904 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:46.319481 11018 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:46.341355 11321 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:46.358831 9978 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:58:46.341874 11333 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260430 07:58:46.342499 10928 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:58:46.343778 11330 log.cc:927] Time spent T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20260430 07:58:46.344856 9929 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1121) took 1767 ms (client timeout 1600 ms). Trace:
I20260430 07:58:46.349304 11299 consensus_queue.cc:1059] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:46.350754 11335 raft_consensus.cc:2749] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679) with higher term. Message: Invalid argument: T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3.
I20260430 07:58:46.320328 11286 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:46.355423 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 79075.1us
I20260430 07:58:46.352600 10998 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:46.327946 11155 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:58:46.413925 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 80517.1us
W20260430 07:58:46.416111 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 108126us
I20260430 07:58:46.338802 11332 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:46.427681 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:46.428022 11332 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:46.428501 11332 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:46.419324 11023 raft_consensus.cc:2804] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:46.353931 11018 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:46.336781 11310 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:46.423136 10534 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:46.324926 11043 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260430 07:58:46.420863 11311 log.cc:927] Time spent T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.356s user 0.000s sys 0.000s
I20260430 07:58:46.432607 11198 consensus_queue.cc:1048] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260430 07:58:46.464303 10928 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:46.468878 11285 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:46.472425 11285 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.457924 11299 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:46.474674 11285 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:46.474520 11299 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.475207 10928 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:46.475747 10928 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:46.474547 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1109) took 1512 ms. Trace:
I20260430 07:58:46.477720 9863 rpcz_store.cc:276] 0430 07:58:44.962469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.233206 (+270737us) service_pool.cc:225] Handling call
0430 07:58:45.233733 (+ 527us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.241857 (+ 8124us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.241897 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.270220 (+ 28323us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.765432 (+495212us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.765465 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.765523 (+ 58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.351878 (+586355us) spinlock_profiling.cc:243] Waited 7.2 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:58:46.474230 (+122352us) raft_consensus.cc:1685] finished
0430 07:58:46.474255 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.474486 (+ 231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14354,"raft.queue_time_us":116,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":7978,"spinlock_wait_cycles":16554240,"thread_start_us":412478,"threads_started":1,"wal-append.queue_time_us":602216}
I20260430 07:58:46.480139 9863 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260430 07:58:46.480734 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1127) took 1061 ms. Trace:
I20260430 07:58:46.482970 11008 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:46.483426 11332 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.483748 11008 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.460711 11043 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:46.481052 9863 rpcz_store.cc:276] 0430 07:58:45.419634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.479336 (+1059702us) service_pool.cc:225] Handling call
0430 07:58:46.479792 (+ 456us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.480502 (+ 710us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.480683 (+ 181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.485677 11043 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:46.486820 11008 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.487207 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:46.487622 10998 raft_consensus.cc:697] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:46.489779 9863 raft_consensus.cc:1275] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:46.490408 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1128) took 1070 ms. Trace:
I20260430 07:58:46.490902 9863 rpcz_store.cc:276] 0430 07:58:45.420008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.489155 (+1069147us) service_pool.cc:225] Handling call
0430 07:58:46.489484 (+ 329us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.490172 (+ 688us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.490355 (+ 183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.424451 9930 raft_consensus.cc:1275] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:46.215713 9932 rpcz_store.cc:276] 0430 07:58:44.407938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.082409 (+674471us) service_pool.cc:225] Handling call
0430 07:58:45.182852 (+100443us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.186808 (+ 3956us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.193050 (+ 6242us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.329630 (+136580us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.777426 (+447796us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.851306 (+ 73880us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.874724 (+ 23418us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.214967 (+340243us) raft_consensus.cc:1685] finished
0430 07:58:46.215000 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.215152 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1514,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":18435,"thread_start_us":345193,"threads_started":1,"wal-append.queue_time_us":591702}
I20260430 07:58:45.761428 9936 rpcz_store.cc:276] 0430 07:58:44.408257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.102559 (+694302us) service_pool.cc:225] Handling call
0430 07:58:45.160361 (+ 57802us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:45.160609 (+ 248us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:45.347376 (+186767us) log.cc:844] Serialized 14 byte log entry
0430 07:58:45.756226 (+408850us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:45.756278 (+ 52us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:45.756316 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.756376 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:45.756515 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15482,"prepare.queue_time_us":233267,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":15616,"thread_start_us":557718,"threads_started":2}
I20260430 07:58:46.330570 11057 consensus_queue.cc:237] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.328747 11122 consensus_queue.cc:237] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.492384 9863 raft_consensus.cc:3055] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:46.501924 9863 raft_consensus.cc:740] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:46.357365 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1125) took 1626 ms. Trace:
I20260430 07:58:46.458673 11310 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:46.463831 11018 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:46.488938 11023 raft_consensus.cc:697] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:46.517412 11092 raft_consensus.cc:3055] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:46.195086 9854 rpcz_store.cc:276] 0430 07:58:44.799572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.834321 (+ 34749us) service_pool.cc:225] Handling call
0430 07:58:44.996377 (+162056us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:44.996631 (+ 254us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:45.366109 (+369478us) log.cc:844] Serialized 14 byte log entry
0430 07:58:46.137551 (+771442us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:46.144976 (+ 7425us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:46.149526 (+ 4550us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.159231 (+ 9705us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.169959 (+ 10728us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19777,"prepare.queue_time_us":375058,"prepare.run_cpu_time_us":3374,"prepare.run_wall_time_us":17693,"raft.queue_time_us":128272,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":4051,"thread_start_us":1070120,"threads_started":3,"wal-append.queue_time_us":741602}
I20260430 07:58:46.489430 11035 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:46.523126 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1116) took 1447 ms. Trace:
I20260430 07:58:46.523967 9863 consensus_queue.cc:260] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.500108 11332 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.519964 9936 raft_consensus.cc:1240] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
W20260430 07:58:46.385875 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:46.568663 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:46.568965 9862 rpcz_store.cc:276] 0430 07:58:45.076006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.695103 (+619097us) service_pool.cc:225] Handling call
0430 07:58:45.695505 (+ 402us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:45.731928 (+ 36423us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:45.731998 (+ 70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:45.747710 (+ 15712us) log.cc:844] Serialized 26 byte log entry
0430 07:58:45.957553 (+209843us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:45.957613 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:45.957713 (+ 100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:46.522842 (+565129us) raft_consensus.cc:1685] finished
0430 07:58:46.522874 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.523050 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":20907,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":9924,"thread_start_us":184305,"threads_started":1,"wal-append.queue_time_us":419800}
I20260430 07:58:46.553807 11008 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:46.375567 9929 rpcz_store.cc:269] 0430 07:58:44.577956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.796351 (+218395us) service_pool.cc:225] Handling call
0430 07:58:46.344612 (+1548261us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54001ac054. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:46.344754 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":730403072}
I20260430 07:58:46.573828 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 90) took 1038 ms. Trace:
I20260430 07:58:46.597077 9936 rpcz_store.cc:276] 0430 07:58:45.535337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.502018 (+966681us) service_pool.cc:225] Handling call
0430 07:58:46.519743 (+ 17725us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.573486 (+ 53743us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.573757 (+ 271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.593982 9863 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:46.511533 11310 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:46.603549 9862 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:46.604562 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1132) took 1183 ms. Trace:
I20260430 07:58:46.605039 9862 rpcz_store.cc:276] 0430 07:58:45.421585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.571498 (+1149913us) service_pool.cc:225] Handling call
0430 07:58:46.603128 (+ 31630us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.604219 (+ 1091us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.604474 (+ 255us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:46.525980 9855 consensus_meta.cc:287] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.859s user 0.013s sys 0.004s
I20260430 07:58:46.372661 11292 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:46.488826 10998 consensus_queue.cc:237] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:46.603499 11093 consensus_meta.cc:287] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.829s user 0.018s sys 0.000s
I20260430 07:58:46.567487 11092 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:46.602948 11299 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.617555 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:46.603021 11286 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:46.618830 9936 raft_consensus.cc:1275] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:58:46.523846 11155 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:46.608470 11226 consensus_meta.cc:287] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.712s user 0.012s sys 0.000s
W20260430 07:58:46.372221 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 77213.9us
I20260430 07:58:46.554066 11061 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:46.547139 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1127) took 1162 ms. Trace:
I20260430 07:58:46.588387 11285 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:46.628371 9862 raft_consensus.cc:2410] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:58:46.603727 11008 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.621474 11006 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.630432 9936 raft_consensus.cc:1275] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:58:46.618009 9855 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:46.502843 9931 rpcz_store.cc:276] 0430 07:58:44.730863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.233827 (+1502964us) service_pool.cc:225] Handling call
0430 07:58:46.285721 (+ 51894us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.307784 (+ 22063us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:46.315317 (+ 7533us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:46.319948 (+ 4631us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:46.325251 (+ 5303us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.331317 (+ 6066us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.353610 (+ 22293us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:46.642565 10566 consensus_meta.cc:287] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.984s user 0.013s sys 0.001s
I20260430 07:58:46.638233 9930 rpcz_store.cc:276] 0430 07:58:45.374609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.284793 (+910184us) service_pool.cc:225] Handling call
0430 07:58:46.333998 (+ 49205us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.506989 (+172991us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.536761 (+ 29772us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.639927 11285 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.648237 11285 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.650364 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:46.652328 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:46.645733 10566 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:46.654520 10566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54770 (request call id 267) took 1212 ms. Trace:
I20260430 07:58:46.655155 10566 rpcz_store.cc:276] 0430 07:58:45.442312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.448114 (+ 5802us) service_pool.cc:225] Handling call
0430 07:58:46.654391 (+1206277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.656785 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1131) took 1236 ms. Trace:
W20260430 07:58:46.657179 11267 raft_consensus.cc:1068] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032: Unable to promote non-voter d66db415f0f0402fa431349afbf59888: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.657975 9855 rpcz_store.cc:276] 0430 07:58:45.420884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.456431 (+ 35547us) service_pool.cc:225] Handling call
0430 07:58:46.656724 (+1200293us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.659674 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1119) took 2092 ms. Trace:
I20260430 07:58:46.660583 10972 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.670760 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1126) took 1297 ms. Trace:
I20260430 07:58:46.632690 11061 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.682327 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1118) took 1598 ms. Trace:
I20260430 07:58:46.673854 10928 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.678251 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1126) took 1277 ms. Trace:
I20260430 07:58:46.720861 9860 rpcz_store.cc:276] 0430 07:58:45.400775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.677217 (+1276442us) service_pool.cc:225] Handling call
0430 07:58:46.677656 (+ 439us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.677890 (+ 234us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:46.677918 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:46.677965 (+ 47us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:46.677999 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.678059 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.678184 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.671259 11092 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.660071 9935 rpcz_store.cc:276] 0430 07:58:44.567022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.706346 (+1139324us) service_pool.cc:225] Handling call
0430 07:58:45.734545 (+ 28199us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.014209 (+279664us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.659398 (+645189us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b54002f1f54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:46.659613 (+ 215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":614021632}
I20260430 07:58:46.633139 11226 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.624572 11093 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.628600 11299 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:58:46.705492 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 277493us
I20260430 07:58:46.716441 11341 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:46.660462 9855 raft_consensus.cc:1275] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:58:46.719678 11340 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:46.719228 9854 raft_consensus.cc:1275] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:58:46.526731 11023 consensus_queue.cc:237] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.638657 11286 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.636608 9863 raft_consensus.cc:1275] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:46.371098 11321 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.714293 9933 rpcz_store.cc:276] 0430 07:58:45.373593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.283206 (+909613us) service_pool.cc:225] Handling call
0430 07:58:46.302875 (+ 19669us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.325872 (+ 22997us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.670292 (+344420us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b54002f1a54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:46.670654 (+ 362us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49719168}
I20260430 07:58:46.798475 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:46.794725 11239 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.504s user 0.057s sys 0.068s
I20260430 07:58:46.811965 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1130) took 1391 ms. Trace:
I20260430 07:58:46.833593 9863 rpcz_store.cc:276] 0430 07:58:45.420565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.491871 (+1071306us) service_pool.cc:225] Handling call
0430 07:58:46.492178 (+ 307us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.811561 (+319383us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.811886 (+ 325us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.835227 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:46.812680 10008 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:46.812500 9828 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:46.814340 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 439594us
I20260430 07:58:46.836050 9863 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:46.829262 10011 raft_consensus.cc:1240] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:46.627573 11292 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.794545 11004 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:46.795253 11006 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:46.807847 11092 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:58:46.853516 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 491863us
I20260430 07:58:46.858198 9863 raft_consensus.cc:1240] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:46.860375 9860 raft_consensus.cc:1275] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:46.860973 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1135) took 1421 ms. Trace:
I20260430 07:58:46.861321 9860 rpcz_store.cc:276] 0430 07:58:45.440067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.847923 (+1407856us) service_pool.cc:225] Handling call
0430 07:58:46.860001 (+ 12078us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.860771 (+ 770us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.860926 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:46.717927 9861 rpcz_store.cc:276] 0430 07:58:45.083697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.238162 (+1154465us) service_pool.cc:225] Handling call
0430 07:58:46.238515 (+ 353us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.238720 (+ 205us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:46.280214 (+ 41494us) log.cc:844] Serialized 14 byte log entry
0430 07:58:46.681676 (+401462us) spinlock_profiling.cc:243] Waited 43.5 ms on lock 0x7b18002a4540. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb566159e1a 00007fb56f7f25c9 00007fb5660181f0 00007fb5661e2b24 00007fb5661e2a95 00007fb56927b440 00007fb56928daf6 00007fb56929e667 00007fb56929e5e8
0430 07:58:46.681909 (+ 233us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:46.681981 (+ 72us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:46.682042 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.682128 (+ 86us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.682259 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16913,"prepare.queue_time_us":87015,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":2867,"raft.queue_time_us":349982,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":141,"spinlock_wait_cycles":100087168,"thread_start_us":115710,"threads_started":2,"wal-append.queue_time_us":339400}
I20260430 07:58:46.827096 9932 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e"
dest_uuid: "77e19ab653f540339c1c613fad1451b5"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:52704
I20260430 07:58:46.872077 9863 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260430 07:58:46.807163 11299 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.847726 11043 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.915184 9863 raft_consensus.cc:3055] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:46.915560 9863 raft_consensus.cc:740] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:46.906885 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:46.908565 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:46.855007 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1134) took 1427 ms. Trace:
W20260430 07:58:46.920989 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 50225.6us
I20260430 07:58:46.864425 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1133) took 1442 ms. Trace:
W20260430 07:58:46.929083 9938 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.007s sys 0.024s
I20260430 07:58:46.870762 11339 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:46.929467 9976 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.030s sys 0.024s
W20260430 07:58:46.927549 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 56210.5us
W20260430 07:58:46.929764 9976 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.030s sys 0.024s
I20260430 07:58:46.850135 11004 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:46.923990 9975 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.015s sys 0.014s
W20260430 07:58:46.931806 9975 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s user 0.015s sys 0.014s
I20260430 07:58:46.931093 9863 consensus_queue.cc:260] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.934397 9861 raft_consensus.cc:1275] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260430 07:58:46.928200 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:46.854130 11006 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.870556 11298 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:46.916496 9932 raft_consensus.cc:606] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
W20260430 07:58:46.937217 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:46.937914 9938 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.602s user 0.008s sys 0.024s
I20260430 07:58:46.958039 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:46.924268 10934 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:46.827167 11341 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:46.929580 11332 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:46.937064 9857 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.011s sys 0.045s
I20260430 07:58:46.915016 11239 consensus_queue.cc:1059] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20260430 07:58:46.957883 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:46.944723 9932 raft_consensus.cc:614] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:58:46.851951 11340 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.918264 11324 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
W20260430 07:58:46.989245 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 59136.3us
I20260430 07:58:46.990170 9828 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:46.921604 11323 consensus_queue.cc:1048] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
W20260430 07:58:46.974018 11349 log.cc:927] Time spent T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.335s user 0.001s sys 0.000s
I20260430 07:58:46.957767 11155 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:46.987776 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1125) took 1610 ms. Trace:
I20260430 07:58:46.951710 11298 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:46.945013 11339 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:47.004340 9978 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:46.916886 9855 rpcz_store.cc:276] 0430 07:58:45.422749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.659544 (+1236795us) service_pool.cc:225] Handling call
0430 07:58:46.659972 (+ 428us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.809798 (+149826us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.850349 (+ 40551us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:46.988704 9857 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.567s user 0.012s sys 0.045s
W20260430 07:58:47.085336 11353 log.cc:927] Time spent T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.210s user 0.000s sys 0.001s
I20260430 07:58:47.218734 11286 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.219491 11043 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:47.223652 11370 log.cc:927] Time spent T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260430 07:58:47.225073 11369 log.cc:927] Time spent T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.220s user 0.000s sys 0.000s
I20260430 07:58:47.225730 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:47.245244 11018 consensus_meta.cc:287] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.698s user 0.008s sys 0.008s
I20260430 07:58:47.194139 11239 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:47.114953 11226 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:47.060344 11093 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:47.086798 11365 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
W20260430 07:58:47.086170 11354 log.cc:927] Time spent T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
I20260430 07:58:46.974638 10934 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:47.238380 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1122) took 1962 ms. Trace:
W20260430 07:58:47.285256 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 338595us
I20260430 07:58:47.123473 11061 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.239470 11299 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:47.243729 11339 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.272143 11292 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:47.031013 9863 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:47.245311 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:47.254035 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:47.201915 11108 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:46.929337 9854 rpcz_store.cc:276] 0430 07:58:45.421926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.598333 (+1176407us) service_pool.cc:225] Handling call
0430 07:58:46.631750 (+ 33417us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.811319 (+179569us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.859134 (+ 47815us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:58:47.403527 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:47.256299 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 298140us
I20260430 07:58:47.269668 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:47.270258 10561 raft_consensus.cc:1240] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:47.270306 11286 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:47.276604 11018 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:47.420028 11286 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:47.420564 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:47.030673 10972 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.212486 9859 rpcz_store.cc:276] 0430 07:58:45.376977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.282957 (+905980us) service_pool.cc:225] Handling call
0430 07:58:46.283212 (+ 255us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.283346 (+ 134us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:46.720194 (+436848us) log.cc:844] Serialized 14 byte log entry
0430 07:58:46.987258 (+267064us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:46.987341 (+ 83us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:46.987418 (+ 77us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.987518 (+ 100us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:46.987695 (+ 177us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":139435,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":2466,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":15977,"thread_start_us":681424,"threads_started":2}
I20260430 07:58:47.015609 11009 raft_consensus.cc:3055] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:47.002959 11008 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.265146 9861 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:47.422161 11018 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.423525 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:47.417114 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1365) took 1430 ms. Trace:
I20260430 07:58:47.276824 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:47.044767 10928 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:47.285328 9858 rpcz_store.cc:276] 0430 07:58:45.275698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.283294 (+1007596us) service_pool.cc:225] Handling call
0430 07:58:46.283583 (+ 289us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:46.283802 (+ 219us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:46.283835 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:46.299892 (+ 16057us) log.cc:844] Serialized 26 byte log entry
0430 07:58:46.809327 (+509435us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:46.809396 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.809494 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:47.238094 (+428600us) raft_consensus.cc:1685] finished
0430 07:58:47.238126 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:47.238298 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":480703,"threads_started":1,"wal-append.queue_time_us":714622}
I20260430 07:58:47.000526 11332 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:47.244779 11004 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.004256 11270 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.276s
I20260430 07:58:47.277223 11006 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.416191 11321 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:47.422318 11286 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.422297 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:47.438515 9859 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:47.424968 9861 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:47.422456 11009 raft_consensus.cc:740] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:47.434906 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:47.436064 11332 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.426249 10008 rpcz_store.cc:276] 0430 07:58:45.986715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.987997 (+ 1282us) service_pool.cc:225] Handling call
0430 07:58:45.988337 (+ 340us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.844988 (+856651us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:47.416882 (+571894us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b54000497d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:47.417055 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1894287616}
I20260430 07:58:47.445863 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:47.446807 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:47.453375 9858 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:47.449090 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:47.449769 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:47.451167 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:47.461949 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:47.462684 10010 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:47.434125 11108 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:47.463521 9828 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:47.465323 11108 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.465479 10009 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260430 07:58:47.465974 11108 raft_consensus.cc:2764] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260430 07:58:47.466853 11345 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:47.467453 11345 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:47.469899 11009 consensus_queue.cc:260] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:47.471350 11009 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:47.475036 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:47.475687 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 471190us
W20260430 07:58:47.480260 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:47.465022 9828 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:47.350924 11339 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:47.425935 10934 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.432297 10560 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260430 07:58:47.485780 9828 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:47.304613 11341 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:47.503592 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 216997us
I20260430 07:58:47.493676 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:37818 (request call id 227) took 2427 ms. Trace:
I20260430 07:58:47.494783 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:47.505690 9828 leader_election.cc:400] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:47.506240 9828 leader_election.cc:403] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260430 07:58:47.279253 11239 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:47.483196 11108 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:47.510596 11323 consensus_queue.cc:1048] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260430 07:58:47.513836 11310 consensus_meta.cc:287] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.896s user 0.009s sys 0.003s
I20260430 07:58:47.514032 11141 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:47.514338 11361 consensus_queue.cc:1048] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:47.516443 11345 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:47.518136 11298 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.495949 11332 consensus_queue.cc:1059] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260430 07:58:47.519917 11341 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:47.520996 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:47.497206 11340 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:47.562305 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:47.504743 9862 rpcz_store.cc:276] 0430 07:58:45.047205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:47.473327 (+2426122us) service_pool.cc:225] Handling call
0430 07:58:47.474710 (+ 1383us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:47.538244 11108 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:47.526813 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:47.457950 11324 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:47.568041 11324 raft_consensus.cc:697] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:47.569126 11324 consensus_queue.cc:237] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:47.531451 9859 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:47.576114 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1364) took 1589 ms. Trace:
I20260430 07:58:47.533059 9863 raft_consensus.cc:1275] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:47.497174 11292 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:47.484620 11321 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:47.671046 9904 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.016s sys 0.038s
I20260430 07:58:47.541461 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:47.544608 11364 log.cc:927] Time spent T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.452s user 0.007s sys 0.000s
I20260430 07:58:47.546351 11198 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:47.525372 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:47.520973 10919 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:47.557252 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:47.562995 10012 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:47.564535 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:47.566826 11108 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:47.528192 9828 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:47.576512 10011 rpcz_store.cc:276] 0430 07:58:45.986452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.987301 (+ 849us) service_pool.cc:225] Handling call
0430 07:58:45.987749 (+ 448us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.840581 (+852832us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:47.575873 (+735292us) spinlock_profiling.cc:243] Waited 840 ms on lock 0x7b54002c5454. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:47.576058 (+ 185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1932944256}
I20260430 07:58:47.578626 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:47.721877 9859 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:47.579165 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:47.663692 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 105290us
I20260430 07:58:47.724766 9863 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:47.539021 11362 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
W20260430 07:58:47.730266 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 202126us
W20260430 07:58:47.673413 9904 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.721s user 0.017s sys 0.038s
W20260430 07:58:47.731434 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:47.558871 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1117) took 2467 ms. Trace:
I20260430 07:58:47.536902 11310 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:47.543282 11339 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
W20260430 07:58:47.686770 9902 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.019s sys 0.026s
W20260430 07:58:47.662860 11335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.918s user 0.024s sys 0.056s
W20260430 07:58:47.742384 9902 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.789s user 0.019s sys 0.027s
I20260430 07:58:47.663297 11292 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:47.705927 11141 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:47.574647 11341 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:47.577906 11122 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.156s
I20260430 07:58:48.010929 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:47.905388 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:47.877840 10003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1362) took 1892 ms. Trace:
W20260430 07:58:47.914757 9938 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.957s user 0.000s sys 0.065s
I20260430 07:58:47.746647 9863 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:58:47.837805 9859 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
W20260430 07:58:47.719139 9827 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.012s sys 0.020s
I20260430 07:58:47.814774 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:47.824399 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:48.079414 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:47.878990 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.020356 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:47.779033 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.079480 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:48.011830 9904 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:48.084069 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:48.078434 9938 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.140s user 0.001s sys 0.076s
W20260430 07:58:48.077306 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:48.079545 9827 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.931s user 0.016s sys 0.029s
I20260430 07:58:48.085457 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:48.086498 9863 raft_consensus.cc:2410] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:58:48.087548 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:48.090327 9859 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:48.091727 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:48.086539 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:48.076844 9862 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:48.114354 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.114722 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.082378 10006 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:58:48.115531 9933 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.171s user 0.001s sys 0.078s
I20260430 07:58:47.714977 11286 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.127930 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:48.128851 9862 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
W20260430 07:58:48.116101 9933 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.171s user 0.001s sys 0.078s
W20260430 07:58:48.056316 11092 consensus_meta.cc:287] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 1.193s user 0.013s sys 0.004s
I20260430 07:58:48.078290 10003 rpcz_store.cc:276] 0430 07:58:45.985787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:45.994959 (+ 9172us) service_pool.cc:225] Handling call
0430 07:58:46.011854 (+ 16895us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:46.118615 (+106761us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:46.118665 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:46.119323 (+ 658us) log.cc:844] Serialized 26 byte log entry
0430 07:58:46.428865 (+309542us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b1800361840. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661c8587 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e5242 00007fb568e1c5ab 00007fb568e220c3 00007fb568e22471 00007fb56923edef
0430 07:58:46.814282 (+385417us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:46.814334 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.814406 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:47.540890 (+726484us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:58:47.877533 (+336643us) raft_consensus.cc:1685] finished
0430 07:58:47.877566 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:47.877748 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113747,"raft.queue_time_us":144893,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":14619,"spinlock_wait_cycles":45085440,"thread_start_us":491406,"threads_started":2,"wal-append.queue_time_us":814395}
W20260430 07:58:47.865424 11352 log.cc:927] Time spent T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.599s user 0.008s sys 0.009s
I20260430 07:58:47.740212 10919 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:48.132243 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:48.044683 11375 log.cc:927] Time spent T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.216s user 0.000s sys 0.000s
I20260430 07:58:48.145819 10919 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:48.076610 11335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.413s user 0.024s sys 0.064s
I20260430 07:58:48.146706 10011 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:58:48.147516 10011 raft_consensus.cc:606] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:58:48.147715 9936 raft_consensus.cc:1240] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
W20260430 07:58:48.001225 11057 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.384s user 0.024s sys 0.057s
I20260430 07:58:47.791033 11108 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election won for term 3
W20260430 07:58:48.054075 11374 log.cc:927] Time spent T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20260430 07:58:48.173655 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.146823 11286 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.175998 11310 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.176663 11310 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:48.177208 11310 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:48.178231 11388 log.cc:927] Time spent T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.316s user 0.001s sys 0.000s
I20260430 07:58:48.178418 11108 raft_consensus.cc:697] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 3 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:48.135277 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:48.126684 10998 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.014s sys 0.020s
W20260430 07:58:47.884835 11376 log.cc:927] Time spent T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.327s user 0.000s sys 0.001s
I20260430 07:58:48.014853 10934 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:47.728943 11298 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:48.089918 11292 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:48.090562 11341 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:47.886445 11340 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.001884 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:48.145999 11339 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.146876 11093 consensus_queue.cc:1048] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20260430 07:58:47.709673 11321 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.015519 11332 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:48.180866 10006 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:58:47.791121 11344 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:47.719584 11198 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:48.159804 11057 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.628s user 0.031s sys 0.063s
I20260430 07:58:48.185876 9862 raft_consensus.cc:2393] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:48.160100 11239 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.176975 11286 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:48.182340 10998 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s user 0.015s sys 0.021s
I20260430 07:58:48.134858 11360 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260430 07:58:48.115530 11141 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.196405 9854 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:48.229740 9862 raft_consensus.cc:1240] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:48.230707 9863 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:48.206504 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 94) took 1566 ms. Trace:
I20260430 07:58:48.188758 11108 consensus_queue.cc:237] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.230732 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 86) took 1661 ms. Trace:
I20260430 07:58:48.242904 9860 rpcz_store.cc:276] 0430 07:58:46.569674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.872554 (+302880us) service_pool.cc:225] Handling call
0430 07:58:46.872900 (+ 346us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:47.114617 (+241717us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:47.114660 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:47.115322 (+ 662us) log.cc:844] Serialized 26 byte log entry
0430 07:58:47.671142 (+555820us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:58:47.671204 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:47.803572 (+132368us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54001bab54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:47.803641 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:48.209787 (+406146us) raft_consensus.cc:1685] finished
0430 07:58:48.209814 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:48.230664 (+ 20850us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":297,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":45053,"spinlock_wait_cycles":520124288,"thread_start_us":407345,"threads_started":1,"wal-append.queue_time_us":573723}
W20260430 07:58:48.205857 9857 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.022s sys 0.056s
W20260430 07:58:48.247171 9857 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.945s user 0.022s sys 0.056s
I20260430 07:58:48.247121 11378 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.248138 11378 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:48.194304 11298 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:48.195482 10934 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.198998 11339 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.196569 11340 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:48.235229 9855 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.990s user 0.020s sys 0.040s
I20260430 07:58:48.236668 11286 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.253455 11057 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:58:48.258227 9759 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.901s user 0.033s sys 0.069s
W20260430 07:58:48.280100 9759 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.924s user 0.034s sys 0.069s
W20260430 07:58:48.267182 10989 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.668s user 0.022s sys 0.029s
W20260430 07:58:48.280679 9901 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.190s user 0.006s sys 0.010s
W20260430 07:58:48.281765 9901 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.191s user 0.006s sys 0.010s
I20260430 07:58:48.287580 9859 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:58:48.288364 9858 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.029s sys 0.034s
W20260430 07:58:48.288785 9858 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s user 0.029s sys 0.034s
W20260430 07:58:48.263995 11351 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.027s sys 0.055s
I20260430 07:58:48.242719 11050 raft_consensus.cc:3055] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:48.232048 11332 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.272558 11379 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:48.273847 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:48.274310 11378 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:48.277805 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1136) took 1914 ms. Trace:
I20260430 07:58:48.345853 9936 rpcz_store.cc:276] 0430 07:58:46.363440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.644498 (+281058us) service_pool.cc:225] Handling call
0430 07:58:46.645786 (+ 1288us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:48.148140 (+1502354us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:48.277503 (+129363us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b540036c9d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:48.277730 (+ 227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3451941760}
I20260430 07:58:48.234596 9929 rpcz_store.cc:276] 0430 07:58:46.640360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.674597 (+ 34237us) service_pool.cc:225] Handling call
0430 07:58:46.674951 (+ 354us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.948495 (+273544us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:46.961617 (+ 13122us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:46.969453 (+ 7836us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:46.974312 (+ 4859us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:48.206165 (+1231853us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b540003b454. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:48.206232 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:48.206424 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":587352448}
W20260430 07:58:48.281083 10989 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.685s user 0.022s sys 0.029s
I20260430 07:58:48.353969 11378 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.354533 11378 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.356755 11378 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.351401 10007 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:48.357659 11378 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.358181 11371 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:48.351115 11285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.318s user 0.030s sys 0.092s
W20260430 07:58:48.359036 11285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.355s user 0.030s sys 0.092s
I20260430 07:58:48.358840 11371 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:48.358246 11378 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.351229 11092 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:48.282928 11324 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.711s user 0.013s sys 0.050s
I20260430 07:58:48.282480 11310 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:48.210088 11344 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:48.349179 10534 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.863s user 0.025s sys 0.052s
W20260430 07:58:48.369968 10534 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.884s user 0.025s sys 0.052s
W20260430 07:58:48.370360 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 883936us
I20260430 07:58:48.330663 11050 raft_consensus.cc:740] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:48.356441 11379 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:48.275245 9855 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.030s user 0.021s sys 0.040s
W20260430 07:58:48.369079 9903 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.015s sys 0.016s
W20260430 07:58:48.388015 9903 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.592s user 0.015s sys 0.016s
W20260430 07:58:48.388582 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 608472us
I20260430 07:58:48.361125 11371 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.394423 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.286037 11286 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.349872 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1131) took 2818 ms. Trace:
I20260430 07:58:48.408684 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.398279 9938 rpcz_store.cc:276] 0430 07:58:45.531121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.318176 (+787055us) service_pool.cc:225] Handling call
0430 07:58:46.318762 (+ 586us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.319005 (+ 243us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:46.938416 (+619411us) log.cc:844] Serialized 14 byte log entry
0430 07:58:48.349130 (+1410714us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:48.349235 (+ 105us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:48.349318 (+ 83us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:48.349534 (+ 216us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:48.349787 (+ 253us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":539233,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":230800,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":9381,"spinlock_wait_cycles":12497280,"thread_start_us":2012250,"threads_started":3,"wal-append.queue_time_us":1403044}
W20260430 07:58:48.332378 11351 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.886s user 0.027s sys 0.056s
I20260430 07:58:48.380355 10534 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:47.742126 9856 rpcz_store.cc:276] 0430 07:58:45.081037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.005800 (+924763us) service_pool.cc:225] Handling call
0430 07:58:46.040488 (+ 34688us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:46.137769 (+ 97281us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:46.142759 (+ 4990us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:46.252130 (+109371us) log.cc:844] Serialized 26 byte log entry
0430 07:58:46.618307 (+366177us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:46.618355 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:46.618427 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:47.532751 (+914324us) raft_consensus.cc:1685] finished
0430 07:58:47.532780 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:47.547971 (+ 15191us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":54940,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":6589,"thread_start_us":353796,"threads_started":2,"wal-append.queue_time_us":563056}
I20260430 07:58:48.420049 10534 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:58:48.367208 11324 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.795s user 0.013s sys 0.051s
I20260430 07:58:48.362378 11378 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.411659 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:48.391089 11386 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:48.455466 10006 raft_consensus.cc:606] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:58:48.414652 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:48.416103 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:48.417518 9861 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.006s sys 0.016s
I20260430 07:58:48.362999 11092 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:48.457857 9861 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s user 0.006s sys 0.016s
I20260430 07:58:48.458551 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.458461 10010 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:48.411370 11321 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.412618 11198 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.409402 11389 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.469508 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 93) took 2261 ms. Trace:
I20260430 07:58:48.470069 9937 rpcz_store.cc:276] 0430 07:58:46.200150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.207295 (+ 7145us) service_pool.cc:225] Handling call
0430 07:58:48.461121 (+2253826us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b540027f6d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:48.461331 (+ 210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4664384768}
I20260430 07:58:48.471941 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.454362 11239 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:48.456027 11018 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election won for term 4
W20260430 07:58:48.465723 11373 log.cc:927] Time spent T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.099s user 0.001s sys 0.001s
I20260430 07:58:48.484567 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.461584 11343 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:48.463815 11393 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:48.464337 11141 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:48.474725 11298 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.522645 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:48.535280 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 95) took 1656 ms. Trace:
I20260430 07:58:48.453066 11050 consensus_queue.cc:260] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:48.543586 11050 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:48.537048 9938 raft_consensus.cc:1240] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260430 07:58:48.505947 11339 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:48.537930 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 525897us
W20260430 07:58:48.535873 9977 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.006s sys 0.017s
W20260430 07:58:48.556464 9977 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.764s user 0.006s sys 0.017s
I20260430 07:58:48.507161 11340 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.501044 11386 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.504478 10934 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:48.534108 10533 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.009s user 0.001s sys 0.027s
W20260430 07:58:48.566452 10533 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.041s user 0.001s sys 0.028s
W20260430 07:58:48.566958 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 1.041e+06us
I20260430 07:58:48.557065 9933 rpcz_store.cc:276] 0430 07:58:46.879339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.879550 (+ 211us) service_pool.cc:225] Handling call
0430 07:58:46.920919 (+ 41369us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.921126 (+ 207us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:58:46.944439 (+ 23313us) log.cc:844] Serialized 14 byte log entry
0430 07:58:48.534827 (+1590388us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:48.534899 (+ 72us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:48.534961 (+ 62us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:48.535053 (+ 92us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:48.535210 (+ 157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8299,"prepare.queue_time_us":25964,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":218647,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":14453,"spinlock_wait_cycles":39256960,"thread_start_us":1589945,"threads_started":2,"wal-append.queue_time_us":1423431}
W20260430 07:58:48.560061 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 834420us
I20260430 07:58:48.480567 11389 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.532858 11332 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:48.562011 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 754728us
W20260430 07:58:48.501444 11322 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.938s user 0.001s sys 0.025s
I20260430 07:58:48.561025 9904 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:48.619755 11389 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.614992 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1382) took 1350 ms. Trace:
I20260430 07:58:48.618341 11286 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.625530 11198 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:48.641113 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.626741 11383 consensus_queue.cc:1059] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:48.627635 9854 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:48.527087 11343 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:48.606534 11322 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.043s user 0.001s sys 0.026s
W20260430 07:58:48.611982 11387 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.150s user 0.000s sys 0.000s
W20260430 07:58:48.606074 11391 log.cc:927] Time spent T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.194s user 0.000s sys 0.000s
W20260430 07:58:48.611001 11384 log.cc:927] Time spent T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260430 07:58:48.630504 9903 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:48.631688 11385 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:48.633033 10010 rpcz_store.cc:276] 0430 07:58:47.265044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:47.463424 (+198380us) service_pool.cc:225] Handling call
0430 07:58:47.463704 (+ 280us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:48.459435 (+995731us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:48.614717 (+155282us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b540037fad4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:48.614935 (+ 218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2286123776}
I20260430 07:58:48.633817 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:58:48.697973 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 64261.4us
W20260430 07:58:48.630543 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 69599.5us
I20260430 07:58:48.700184 9904 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:58:48.618188 11292 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.647061 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.647504 10005 raft_consensus.cc:1240] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:48.647259 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:48.687891 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:48.716976 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:48.723361 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 92933.1us
I20260430 07:58:48.641033 11306 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:48.532406 11393 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:48.725687 9904 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:48.648358 11381 log.cc:927] Time spent T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.231s user 0.000s sys 0.001s
I20260430 07:58:48.642266 11340 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:48.642814 11360 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.731662 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:48.746524 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.730023 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1124) took 3379 ms. Trace:
I20260430 07:58:48.651561 11344 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:48.762975 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.575109 11386 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.766079 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.766933 9903 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:48.729388 11341 consensus_meta.cc:287] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.537s user 0.007s sys 0.004s
I20260430 07:58:48.698264 11057 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:48.767846 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:48.767949 9827 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s user 0.005s sys 0.014s
I20260430 07:58:48.643357 11351 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.099s
I20260430 07:58:48.754540 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.774618 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.727928 11389 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:58:48.776306 9827 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.670s user 0.005s sys 0.014s
I20260430 07:58:48.776338 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.722595 11242 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:48.645555 11198 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.788692 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.743124 11301 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:48.733635 11332 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:48.764441 11108 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:58:48.783867 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 58258.6us
I20260430 07:58:48.734843 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:48.786855 9903 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:48.787684 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.790849 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:48.862785 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 75977.9us
I20260430 07:58:48.863070 9859 raft_consensus.cc:2468] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
W20260430 07:58:48.783031 9828 connection.cc:254] RPC call timeout handler was delayed by -3.2248071509999932s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1837s for a response
I20260430 07:58:48.840492 10008 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:58:48.864284 9903 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:48.864979 10008 raft_consensus.cc:606] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:58:48.769609 11339 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:48.851073 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.853237 9904 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:48.722932 11338 consensus_queue.cc:1048] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
W20260430 07:58:48.864305 9828 connection.cc:254] RPC call timeout handler was delayed by -3.1924071509999976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18730000000000002s for a response
I20260430 07:58:48.869057 9929 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:48.870121 11400 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:48.876937 9828 connection.cc:254] RPC call timeout handler was delayed by -3.1105071509999789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19640000000000002s for a response
I20260430 07:58:48.753796 11371 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.087s
I20260430 07:58:48.729992 11340 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:48.880716 9904 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:48.792296 11301 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:48.882251 10012 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:48.764153 11378 consensus_queue.cc:1048] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20260430 07:58:48.791923 11310 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:48.768977 11323 consensus_queue.cc:1048] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20260430 07:58:48.891235 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260430 07:58:48.926002 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.860733 11406 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:48.937386 11292 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:48.881893 9828 connection.cc:254] RPC call timeout handler was delayed by -2.8648071509999795s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2237s for a response
I20260430 07:58:48.937776 11406 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:58:48.938068 9828 connection.cc:254] RPC call timeout handler was delayed by -2.5093071509999731s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26320000000000005s for a response
I20260430 07:58:48.938284 11406 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:48.919167 11400 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.939208 11393 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.939761 11393 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:48.940215 11393 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:48.940965 11400 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:48.942226 10007 consensus_meta.cc:287] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.584s user 0.007s sys 0.003s
I20260430 07:58:48.942351 11400 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260430 07:58:48.942515 11395 log.cc:927] Time spent T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260430 07:58:48.942665 10007 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:48.893797 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1310) took 1445 ms. Trace:
I20260430 07:58:48.778131 11138 consensus_queue.cc:1048] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.102s
I20260430 07:58:48.850425 11332 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:58:48.839660 11379 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:48.784860 11362 consensus_queue.cc:1048] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.097s
I20260430 07:58:48.998323 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:48.998541 11410 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:48.999080 11410 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.997956 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.000211 10568 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:48.778262 11341 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.787034 11023 consensus_queue.cc:1059] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.104s
I20260430 07:58:49.001458 11023 raft_consensus.cc:3055] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:48.882472 11239 raft_consensus.cc:697] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:49.001860 11023 raft_consensus.cc:740] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:49.002413 11341 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.002684 11023 consensus_queue.cc:260] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.004068 11023 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:49.004405 11332 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:49.004534 11393 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.893762 11394 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:48.750166 11306 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.005465 11394 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:48.782466 11057 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.840821 11298 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:49.011474 11298 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.013497 11298 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:48.934321 11396 log.cc:927] Time spent T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.155s user 0.000s sys 0.000s
I20260430 07:58:48.936168 10004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 70) took 1946 ms. Trace:
I20260430 07:58:49.015955 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.015416 10004 rpcz_store.cc:276] 0430 07:58:46.990047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.990209 (+ 162us) service_pool.cc:225] Handling call
0430 07:58:47.795551 (+805342us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b58000a2098. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:47.795766 (+ 215us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:47.812906 (+ 17140us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:47.812935 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:47.829318 (+ 16383us) log.cc:844] Serialized 26 byte log entry
0430 07:58:48.522491 (+693173us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:58:48.522530 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:48.522598 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:48.935862 (+413264us) raft_consensus.cc:1685] finished
0430 07:58:48.935885 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:48.936102 (+ 217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16708,"raft.queue_time_us":17324,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":6452,"spinlock_wait_cycles":91617536,"thread_start_us":375015,"threads_started":1,"wal-append.queue_time_us":833392}
I20260430 07:58:48.936398 11198 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.883069 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:48.885406 11302 raft_consensus.cc:2749] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:48.861756 11285 consensus_queue.cc:1059] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
I20260430 07:58:48.885653 11339 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.942833 11400 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:48.751384 11360 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:48.899051 9904 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:49.032580 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 348742us
W20260430 07:58:49.033699 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 169409us
I20260430 07:58:48.922899 11310 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:48.983660 10003 raft_consensus.cc:1240] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:48.987452 11379 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:48.981535 11338 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:48.991047 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.046417 10012 raft_consensus.cc:2393] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:48.991442 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.991726 11122 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:48.992446 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.993814 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:48.996591 11389 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:48.998044 11323 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:49.002840 11239 consensus_queue.cc:237] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.075842 11386 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:48.928037 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:58:49.009943 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.010632 11306 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.079232 9854 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20260430 07:58:49.080567 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:48.749089 9857 rpcz_store.cc:276] 0430 07:58:45.350782 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.284140 (+933358us) service_pool.cc:225] Handling call
0430 07:58:46.320863 (+ 36723us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:46.335947 (+ 15084us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:47.276500 (+940553us) log.cc:844] Serialized 14 byte log entry
0430 07:58:48.642867 (+1366367us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:48.646959 (+ 4092us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:48.677074 (+ 30115us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:48.720689 (+ 43615us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:48.729791 (+ 9102us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":136852,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":304919,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":5008,"thread_start_us":2178872,"threads_started":3,"wal-append.queue_time_us":1122575}
W20260430 07:58:49.082176 11390 log.cc:927] Time spent T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.557s user 0.004s sys 0.002s
I20260430 07:58:49.043752 11310 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:48.945176 9937 raft_consensus.cc:2410] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260430 07:58:48.985862 9858 rpcz_store.cc:276] 0430 07:58:47.448308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:47.454224 (+ 5916us) service_pool.cc:225] Handling call
0430 07:58:47.454624 (+ 400us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:47.454897 (+ 273us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:47.454931 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:47.485223 (+ 30292us) log.cc:844] Serialized 26 byte log entry
0430 07:58:48.289584 (+804361us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:48.783723 (+494139us) log.cc:844] Serialized 14 byte log entry
0430 07:58:48.784089 (+ 366us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:48.784193 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:48.893487 (+109294us) raft_consensus.cc:1685] finished
0430 07:58:48.893517 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:48.893718 (+ 201us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1219779,"threads_started":2,"wal-append.queue_time_us":1030722}
I20260430 07:58:49.037433 11292 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:49.012887 11057 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.084769 11292 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:49.020797 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:48.872817 11108 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:49.018631 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 78473.4us
I20260430 07:58:49.089625 9828 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:49.090979 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:48.916384 10934 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:49.093055 10934 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:58:49.007441 10007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1371) took 2074 ms (client timeout 2111 ms). Trace:
I20260430 07:58:48.924032 11301 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:49.031679 11400 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:49.047869 9859 raft_consensus.cc:1240] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260430 07:58:49.048246 9860 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:49.048341 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:49.113029 9863 raft_consensus.cc:2410] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260430 07:58:49.033984 11402 raft_consensus.cc:3055] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:49.054019 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.056525 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.035655 11285 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:58:49.031762 9856 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:49.050699 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:49.062885 10561 raft_consensus.cc:1275] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260430 07:58:49.062775 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.062819 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.063715 9977 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:49.040261 11341 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:49.067198 11343 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:49.065689 11389 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.044953 11379 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:49.056095 11338 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.069717 11324 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:49.060492 11122 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.071652 11323 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.074186 11410 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.083168 11401 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:49.094126 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:48.901768 11018 raft_consensus.cc:697] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:49.093518 10934 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:49.100600 10007 rpcz_store.cc:269] 0430 07:58:46.932806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.949444 (+ 16638us) service_pool.cc:225] Handling call
0430 07:58:48.273648 (+1324204us) spinlock_profiling.cc:243] Waited 5.72 ms on lock 0x7b58000a2098. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:58:48.985516 (+711868us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13158272}
I20260430 07:58:49.101161 11108 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:49.105099 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.107013 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.186728 11379 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:49.136585 9903 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.131:39205 timed out after 2.111s (SENT) [suppressed 4 similar messages]
I20260430 07:58:49.157970 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:49.134693 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:49.080605 11092 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.164597 9828 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:49.101186 11286 raft_consensus.cc:3055] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Stepping down as leader of term 2
W20260430 07:58:49.165666 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 101970us
I20260430 07:58:49.171005 9862 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
W20260430 07:58:49.214896 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 50358.2us
I20260430 07:58:49.211047 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.189997 11403 consensus_queue.cc:1059] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260430 07:58:49.177498 11341 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:49.168793 11093 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:49.084851 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:49.113960 11321 raft_consensus.cc:2749] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:49.152287 11340 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:49.113883 11402 raft_consensus.cc:740] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:58:49.190428 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 291284us
W20260430 07:58:49.111572 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 56100.6us
W20260430 07:58:49.208806 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 72193.4us
I20260430 07:58:49.161705 11386 consensus_queue.cc:1059] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260430 07:58:49.168534 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.488229 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.460803 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:44006 (request call id 463) took 1228 ms. Trace:
I20260430 07:58:49.467244 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.207305 9860 raft_consensus.cc:1275] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:49.474339 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:58:49.535276 9904 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:49.535514 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:49.537242 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:49.241849 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.404239 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.486919 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.532481 10009 rpcz_store.cc:276] 0430 07:58:48.201734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:48.261798 (+ 60064us) service_pool.cc:225] Handling call
0430 07:58:49.406593 (+1144795us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54000cce54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ed470 00007fb56928c31a 00007fb56e25b11e 00007fb5677d7437 00007fb5677d73c2 00007fb5677d72f1 00007fb5677d726c 00007fb5677d6522 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:49.430365 (+ 23772us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880236288}
I20260430 07:58:49.428990 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:49.542450 10533 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:49.227489 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.406828 10534 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:49.539286 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:49.458803 9828 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:58:49.562145 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 155333us
I20260430 07:58:49.219434 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.560632 9904 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:49.565001 9933 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260430 07:58:49.565816 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:49.564714 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.567924 9938 raft_consensus.cc:1275] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260430 07:58:49.558197 9860 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:49.568742 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:49.378734 11385 consensus_meta.cc:287] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.538s user 0.009s sys 0.003s
W20260430 07:58:49.474777 11372 log.cc:927] Time spent T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 1.195s user 0.006s sys 0.000s
I20260430 07:58:49.447570 11360 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:49.254576 11323 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:49.383112 11423 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:49.606434 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:49.257771 11394 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.227648 11424 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:49.606312 11423 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.227298 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.207851 11292 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.574455 10533 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:49.557231 10566 raft_consensus.cc:1240] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:49.556795 11404 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:49.195510 11324 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.623636 11423 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.588217 11429 consensus_queue.cc:1048] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260430 07:58:49.626027 11324 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:49.241946 11339 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.593369 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:49.564810 11408 raft_consensus.cc:3055] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:49.595479 11057 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:58:49.627302 11408 raft_consensus.cc:740] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:49.628621 11408 consensus_queue.cc:260] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.616142 9854 raft_consensus.cc:1275] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260430 07:58:49.630416 11408 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:49.474433 11338 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:49.633077 9933 raft_consensus.cc:1240] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:49.633744 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:49.623647 11404 raft_consensus.cc:697] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:49.634358 9938 raft_consensus.cc:2376] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3. Re-sending same reply.
I20260430 07:58:49.593322 11403 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:49.515851 11410 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:49.531463 11286 raft_consensus.cc:740] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:49.204322 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1301) took 2286 ms. Trace:
I20260430 07:58:49.596508 9863 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:49.597589 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.597662 9904 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:49.644153 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:49.645097 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 47487.5us
I20260430 07:58:49.646003 10534 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:49.646211 9904 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:49.590049 9860 raft_consensus.cc:1240] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:49.586251 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.538455 11023 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:49.475405 11389 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:49.576099 11418 raft_consensus.cc:2804] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:49.491550 11400 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.249200 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:49.537945 11093 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:49.615244 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:49.663585 9937 raft_consensus.cc:2393] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 2: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:49.574723 11417 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:49.481691 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.590926 11385 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:49.621058 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 50288.1us
I20260430 07:58:49.621913 11394 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:49.467810 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.259029 11406 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.476341 11332 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.644966 9863 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:49.648924 9860 raft_consensus.cc:1275] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:49.631979 11301 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.656325 11239 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:49.650956 9904 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:49.706370 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 55487.8us
I20260430 07:58:49.655961 11418 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:49.661514 9938 raft_consensus.cc:1275] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260430 07:58:49.641887 11377 log.cc:927] Time spent T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.129s user 0.000s sys 0.001s
I20260430 07:58:49.474576 11018 consensus_queue.cc:237] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.642975 11410 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.688531 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.727543 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.689465 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:49.656598 11310 consensus_meta.cc:287] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.570s user 0.010s sys 0.001s
I20260430 07:58:49.695760 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.653163 11285 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:58:49.649118 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:49.619179 11343 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20260430 07:58:49.635283 11379 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.707158 9904 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:49.205350 11401 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.652526 9936 raft_consensus.cc:1240] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:49.753810 9929 raft_consensus.cc:2393] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:49.753973 11401 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.707695 11286 consensus_queue.cc:260] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.708443 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:49.710644 11432 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260430 07:58:49.710911 9931 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:49.670146 11093 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:49.718585 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:49.673915 9937 raft_consensus.cc:1275] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260430 07:58:49.755888 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 322713us
I20260430 07:58:49.767216 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:49.720280 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.635246 11404 consensus_queue.cc:237] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:49.735013 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 276183us
I20260430 07:58:49.770798 9828 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:49.695645 11394 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:49.658180 11023 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:49.720080 11418 raft_consensus.cc:3060] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:49.737485 11310 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.795277 11310 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.750214 9904 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:49.750716 10533 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:49.785629 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 173627us
W20260430 07:58:49.796808 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 46142.1us
I20260430 07:58:49.707762 11057 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:49.766144 9938 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
W20260430 07:58:49.799345 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 49203us
I20260430 07:58:49.672428 11402 consensus_queue.cc:260] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.711604 10934 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:49.780668 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:49.783380 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.783632 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.785712 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:49.809369 9937 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:49.740813 11337 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:49.786275 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.755712 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:49.787442 11403 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:49.788096 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:49.789595 11400 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:49.755304 11092 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:49.790907 11410 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:49.791841 11436 log.cc:927] Time spent T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260430 07:58:49.642956 9855 rpcz_store.cc:276] 0430 07:58:46.918015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:47.244654 (+326639us) service_pool.cc:225] Handling call
0430 07:58:47.245028 (+ 374us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:48.401133 (+1156105us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:48.408703 (+ 7570us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:48.487168 (+ 78465us) log.cc:844] Serialized 26 byte log entry
0430 07:58:48.919615 (+432447us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:49.113853 (+194238us) log.cc:844] Serialized 14 byte log entry
0430 07:58:49.169568 (+ 55715us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:49.176515 (+ 6947us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:49.186056 (+ 9541us) raft_consensus.cc:1685] finished
0430 07:58:49.191054 (+ 4998us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:49.197078 (+ 6024us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2564,"prepare.queue_time_us":89604,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":195451,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":21969,"thread_start_us":1580382,"threads_started":3,"wal-append.queue_time_us":516001}
I20260430 07:58:49.769261 11341 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:49.795928 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:49.800390 9904 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:49.886598 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 86323.6us
I20260430 07:58:49.888095 9904 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:49.802526 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.808885 11383 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:49.836849 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:49.784734 11285 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:49.786491 11394 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:49.786831 11401 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:49.843020 11286 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:49.872138 10533 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:49.802066 11423 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:49.774941 11385 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:49.890113 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:49.911293 9938 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:49.890861 9904 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:49.912384 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:49.893126 9937 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:49.914158 9937 raft_consensus.cc:1240] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260430 07:58:49.916045 9937 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:58:49.842552 11402 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:49.917582 11414 raft_consensus.cc:2804] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:49.918048 11414 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:49.918435 11414 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:49.851642 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:49.907253 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:49.907716 10008 raft_consensus.cc:1240] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:49.917452 11429 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:49.922623 11429 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:49.923242 11429 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:49.923410 9904 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:49.925688 11057 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:49.926795 10533 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:49.928095 11423 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:49.945132 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:49.838732 11337 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.895572 11383 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:49.950627 11429 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.955026 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:49.897385 11310 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:49.897920 11400 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:49.893707 11285 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:49.967211 11198 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.785s user 0.020s sys 0.035s
I20260430 07:58:49.903048 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.949035 11410 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:49.902514 11386 consensus_queue.cc:1059] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260430 07:58:49.949726 11417 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:49.948772 11401 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:49.960036 11383 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:49.994804 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:49.959873 11440 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:49.967953 11400 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:58:50.015369 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:49.978643 11198 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.810s user 0.020s sys 0.035s
I20260430 07:58:49.892081 11403 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:49.960600 11341 consensus_queue.cc:1059] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260430 07:58:49.963145 11310 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:50.028735 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:49.967296 11023 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:50.033054 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 108035us
I20260430 07:58:49.997002 11301 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:49.997025 9930 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:50.004673 11406 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.045449 9828 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:50.031802 11400 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:50.049571 9862 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:50.041728 11310 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:50.036130 9931 raft_consensus.cc:1240] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260430 07:58:50.032207 11440 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.032456 11057 raft_consensus.cc:697] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:50.069736 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.023633 11401 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.005564 11410 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.062979 11285 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:50.075598 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.053875 11442 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:50.109047 9863 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:50.070468 10934 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.100780 11301 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:50.113587 9860 raft_consensus.cc:1275] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:58:50.070609 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.131299 9857 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:50.132944 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.134667 11413 raft_consensus.cc:3055] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:50.116420 11414 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.135213 11413 raft_consensus.cc:740] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:50.074353 11402 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:50.163172 10008 raft_consensus.cc:3055] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:50.165222 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:50.165669 9935 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260430 07:58:50.163676 10008 raft_consensus.cc:740] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:50.170246 10565 raft_consensus.cc:1240] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260430 07:58:50.170377 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.171222 9863 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
W20260430 07:58:50.172669 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:50.128008 9855 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:50.174722 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.170060 10008 consensus_queue.cc:260] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.180418 10008 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:50.181463 9863 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:50.180691 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.186203 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.135627 11298 consensus_queue.cc:1059] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260430 07:58:50.139127 11301 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.188818 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.191434 9937 raft_consensus.cc:2393] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:50.190259 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:50.193222 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.181530 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:50.197705 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:50.187042 11341 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:50.199337 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.209632 9863 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:50.200736 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1142) took 1151 ms. Trace:
W20260430 07:58:50.201552 11445 log.cc:927] Time spent T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20260430 07:58:50.222520 10568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54778 (request call id 86) took 1221 ms. Trace:
I20260430 07:58:50.222991 10568 rpcz_store.cc:276] 0430 07:58:49.007391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.009641 (+ 2250us) service_pool.cc:225] Handling call
0430 07:58:50.222267 (+1212626us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b5400430154. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:50.222451 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2201804672}
I20260430 07:58:50.224031 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1151) took 1164 ms. Trace:
I20260430 07:58:50.224179 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 71) took 3233 ms. Trace:
I20260430 07:58:50.224282 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.224481 10005 rpcz_store.cc:276] 0430 07:58:46.990373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:46.990502 (+ 129us) service_pool.cc:225] Handling call
0430 07:58:47.272582 (+282080us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:48.701381 (+1428799us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.223952 (+1522571us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000cce54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:50.224105 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3160032000}
I20260430 07:58:50.225437 9929 raft_consensus.cc:2410] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260430 07:58:50.226688 11414 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.190382 11270 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:50.228032 11270 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.229447 9931 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260430 07:58:50.230149 11270 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:50.232081 9854 rpcz_store.cc:276] 0430 07:58:49.049699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.092381 (+ 42682us) service_pool.cc:225] Handling call
0430 07:58:49.092985 (+ 604us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:49.630303 (+537318us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.200407 (+570104us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b54001bb554. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:50.200673 (+ 266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1138635648}
I20260430 07:58:50.187280 11057 consensus_queue.cc:237] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.234037 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.235338 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:50.235932 9854 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:50.177423 11402 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.237895 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.238605 9929 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:50.238915 11402 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:50.204192 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:50.205469 11417 raft_consensus.cc:3055] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:50.241207 11417 raft_consensus.cc:740] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:50.241287 10564 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:50.242189 11417 consensus_queue.cc:260] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.213186 11301 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:50.230827 11410 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.243466 11417 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:50.243660 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.205890 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.246876 11337 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:50.206351 11093 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.248732 11332 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.249148 11093 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:50.197590 11413 consensus_queue.cc:260] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.250622 11413 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:50.186570 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.224447 9859 rpcz_store.cc:276] 0430 07:58:49.059680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.594935 (+535255us) service_pool.cc:225] Handling call
0430 07:58:49.600210 (+ 5275us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:49.813453 (+213243us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:49.813502 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:49.836676 (+ 23174us) log.cc:844] Serialized 26 byte log entry
0430 07:58:50.034411 (+197735us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:50.034456 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.034524 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:50.223850 (+189326us) raft_consensus.cc:1685] finished
0430 07:58:50.223871 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:50.223981 (+ 110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5777,"raft.queue_time_us":19659,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":5771,"thread_start_us":326948,"threads_started":2,"wal-append.queue_time_us":297175}
I20260430 07:58:50.253710 9857 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:50.254418 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:50.254777 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.255980 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.256129 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:50.256502 9854 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:58:50.256631 9859 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:50.206310 11403 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.210567 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:50.257647 10012 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:50.184655 11429 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.258092 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.186774 11122 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:50.212136 11341 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.259357 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.230798 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:50.261258 11341 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.213109 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:50.184139 11418 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.262018 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.215137 9930 raft_consensus.cc:2393] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:50.217208 11394 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.280200 10559 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:50.282120 11394 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.203570 11440 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.249578 10003 raft_consensus.cc:1240] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260430 07:58:50.256774 9857 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:50.283349 11270 consensus_queue.cc:1059] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:50.207976 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1141) took 1158 ms. Trace:
I20260430 07:58:50.266633 10008 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260430 07:58:50.266520 10533 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:50.273240 10560 raft_consensus.cc:1240] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:50.274456 11418 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.274490 9930 raft_consensus.cc:1240] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:50.230227 11242 raft_consensus.cc:3055] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:50.275103 9859 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:50.278945 11400 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.284879 9858 rpcz_store.cc:276] 0430 07:58:49.049360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.092059 (+ 42699us) service_pool.cc:225] Handling call
0430 07:58:49.092364 (+ 305us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:49.092582 (+ 218us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:49.111686 (+ 19104us) log.cc:844] Serialized 14 byte log entry
0430 07:58:50.204765 (+1093079us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b18002ec960. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb568e2a85c 00007fb5661c8c0b 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e5242 00007fb568e1c5ab 00007fb568e220c3 00007fb568e225da
0430 07:58:50.204973 (+ 208us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:50.205033 (+ 60us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:50.205080 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.205149 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:50.207913 (+ 2764us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29500,"prepare.queue_time_us":21434,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":5096,"raft.queue_time_us":534,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":7087488,"thread_start_us":94649,"threads_started":1,"wal-append.queue_time_us":482856}
I20260430 07:58:50.339047 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:50.339655 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:50.289562 9977 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:50.262838 9854 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
W20260430 07:58:50.311861 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 48177.3us
I20260430 07:58:50.312433 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.291061 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.339794 9858 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:50.275640 11310 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:50.307327 11414 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:58:50.308132 11323 consensus_queue.cc:1059] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260430 07:58:50.348107 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:50.355933 9859 raft_consensus.cc:3060] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:50.335675 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:50.354760 9934 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:50.316061 11122 consensus_queue.cc:1048] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260430 07:58:50.315836 11298 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260430 07:58:50.341549 11270 raft_consensus.cc:3055] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:50.340973 11242 raft_consensus.cc:740] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:50.334299 9863 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:58:50.355684 11337 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:50.368227 9858 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:58:50.368717 9938 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
W20260430 07:58:50.361598 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 131993us
I20260430 07:58:50.343658 11383 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.372982 11337 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:50.395707 10533 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:50.387799 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:50.407279 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.382411 11242 consensus_queue.cc:260] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.406032 11337 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:50.414286 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.414969 9936 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:50.391381 11400 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.409526 11298 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:50.410429 11323 raft_consensus.cc:3055] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:50.410717 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.412398 9978 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:50.424057 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:50.424535 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:50.425706 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:50.388464 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1303) took 3384 ms. Trace:
I20260430 07:58:50.416998 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:50.417969 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.420409 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:50.418231 9859 raft_consensus.cc:2468] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:50.422322 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.413225 11383 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.427464 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.428315 10534 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:50.429041 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.429684 9858 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:50.430860 11414 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:50.430913 11440 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:50.431627 11414 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:50.431710 11440 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:50.432240 11440 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:50.433424 11341 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:50.434479 11341 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.436096 11122 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:50.436583 11122 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:50.436743 11341 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:50.437011 11122 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:50.437333 9929 raft_consensus.cc:1240] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260430 07:58:50.437439 9855 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 1.
I20260430 07:58:50.374898 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.438271 11338 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:50.438584 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.369494 11270 raft_consensus.cc:740] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:50.440021 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 94) took 1031 ms. Trace:
I20260430 07:58:50.368700 11343 consensus_queue.cc:1059] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.055s
I20260430 07:58:50.440189 11239 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:50.432164 11414 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:50.386261 9930 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260430 07:58:50.441067 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:50.451715 9858 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:58:50.441644 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 152081us
I20260430 07:58:50.442337 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:50.426918 9861 rpcz_store.cc:276] 0430 07:58:47.003710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:47.452696 (+448986us) service_pool.cc:225] Handling call
0430 07:58:47.496953 (+ 44257us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:47.572691 (+ 75738us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:47.572726 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:47.573165 (+ 439us) log.cc:844] Serialized 26 byte log entry
0430 07:58:48.458424 (+885259us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:48.776764 (+318340us) log.cc:844] Serialized 14 byte log entry
0430 07:58:48.784956 (+ 8192us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.388063 (+1603107us) spinlock_profiling.cc:243] Waited 75.3 ms on lock 0x7b5400207e54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:50.388109 (+ 46us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:50.388152 (+ 43us) raft_consensus.cc:1685] finished
0430 07:58:50.388261 (+ 109us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:50.388407 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7673,"prepare.queue_time_us":558179,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":227527,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":43420,"spinlock_wait_cycles":173291904,"thread_start_us":1029067,"threads_started":2,"wal-append.queue_time_us":978332,"wal-append.run_cpu_time_us":3418,"wal-append.run_wall_time_us":1330974}
I20260430 07:58:50.428048 9862 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:50.447474 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:50.455513 9977 leader_election.cc:336] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.600s (SENT)
I20260430 07:58:50.370505 11336 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.075s
I20260430 07:58:50.458513 9862 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260430 07:58:50.423000 11323 raft_consensus.cc:740] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:50.464040 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:50.467890 10565 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:50.465391 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:50.440431 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 95) took 1031 ms. Trace:
I20260430 07:58:50.426225 11401 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.451480 9930 pending_rounds.cc:85] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 4
I20260430 07:58:50.456874 11343 raft_consensus.cc:3055] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:50.444867 11239 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:58:50.480167 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:50.452107 10928 raft_consensus.cc:2804] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:50.450670 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:50.504740 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:50.468678 9862 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:58:50.466300 11242 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:50.522473 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 98145.1us
I20260430 07:58:50.560320 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1156) took 1377 ms. Trace:
I20260430 07:58:50.563581 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.560796 9860 rpcz_store.cc:276] 0430 07:58:49.182674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.705116 (+522442us) service_pool.cc:225] Handling call
0430 07:58:49.706577 (+ 1461us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:50.125236 (+418659us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.560045 (+434809us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54001b9c54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:50.560247 (+ 202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":934413056}
I20260430 07:58:50.568012 9860 raft_consensus.cc:1275] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:50.574306 9860 raft_consensus.cc:1275] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:50.440299 11270 consensus_queue.cc:260] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.510529 9930 pending_rounds.cc:107] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260430 07:58:50.504033 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:50.510386 11343 raft_consensus.cc:740] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:50.621961 9978 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:50.470434 11393 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.114s
I20260430 07:58:50.646848 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:50.613924 10928 raft_consensus.cc:697] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:50.445602 9854 rpcz_store.cc:276] 0430 07:58:49.408820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.439167 (+1030347us) service_pool.cc:225] Handling call
0430 07:58:50.439477 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:50.439658 (+ 181us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:50.439693 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:50.439741 (+ 48us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:50.439781 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.439842 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:50.439962 (+ 120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:50.620175 9930 raft_consensus.cc:2889] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:50.651690 10007 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
W20260430 07:58:50.693790 9828 leader_election.cc:336] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.966s (SENT)
I20260430 07:58:50.694304 9828 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:50.582187 11468 consensus_queue.cc:1059] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.154s
I20260430 07:58:50.646634 11413 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:50.583671 11023 consensus_queue.cc:1059] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.071s
I20260430 07:58:50.657867 11386 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:50.634034 11323 consensus_queue.cc:260] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.677376 11270 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:50.491243 9855 rpcz_store.cc:276] 0430 07:58:49.409154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.439611 (+1030457us) service_pool.cc:225] Handling call
0430 07:58:50.439919 (+ 308us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:50.440083 (+ 164us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:50.440111 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:50.440168 (+ 57us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:50.440209 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.440269 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:50.440378 (+ 109us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:50.725847 11413 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.742492 11413 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.711397 11403 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:50.710281 11470 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:50.760352 9860 raft_consensus.cc:1240] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:50.739223 9857 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:50.754663 11323 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:50.769924 9858 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
W20260430 07:58:50.774047 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 64255.8us
I20260430 07:58:50.785885 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1414) took 1673 ms. Trace:
I20260430 07:58:50.786381 10011 rpcz_store.cc:276] 0430 07:58:49.112045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.578852 (+466807us) service_pool.cc:225] Handling call
0430 07:58:49.593441 (+ 14589us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:49.877533 (+284092us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:49.877579 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:49.877640 (+ 61us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:49.877683 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.785620 (+907937us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b540004a1d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:50.785669 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:50.785812 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":568056064}
I20260430 07:58:50.789626 11400 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:50.790169 11400 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:50.771239 11465 log.cc:927] Time spent T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.280s user 0.001s sys 0.000s
I20260430 07:58:50.792207 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.771405 9860 raft_consensus.cc:1240] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:50.795084 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.795342 11400 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.796031 9932 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:50.799072 11310 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.763103 9854 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260430 07:58:50.801657 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.723977 11343 consensus_queue.cc:260] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:50.775265 11466 log.cc:927] Time spent T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
I20260430 07:58:50.813896 9936 raft_consensus.cc:1275] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:58:50.816228 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.813192 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.818636 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 92) took 1410 ms. Trace:
I20260430 07:58:50.819042 9856 rpcz_store.cc:276] 0430 07:58:49.408074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.790785 (+382711us) service_pool.cc:225] Handling call
0430 07:58:49.791140 (+ 355us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:49.929533 (+138393us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:58:49.929844 (+ 311us) log.cc:844] Serialized 14 byte log entry
0430 07:58:50.066712 (+136868us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:50.076609 (+ 9897us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:50.108689 (+ 32080us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.817921 (+709232us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b180006fc40. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661b3be6 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9
0430 07:58:50.818425 (+ 504us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54001bab54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:50.818460 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:50.818577 (+ 117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2777,"prepare.queue_time_us":623,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":12760,"raft.run_cpu_time_us":7805,"raft.run_wall_time_us":50494,"spinlock_wait_cycles":396931840,"thread_start_us":123189,"threads_started":1,"wal-append.queue_time_us":234760}
I20260430 07:58:50.821358 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:50.822355 11385 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:50.822867 11385 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:50.823154 11400 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260430 07:58:50.825538 9828 leader_election.cc:336] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.077s (SENT)
I20260430 07:58:50.825985 9828 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:50.821425 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.827634 9931 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:50.827875 9903 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:50.828652 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:50.829468 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:50.829710 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.829946 11050 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:50.831051 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:50.829970 11337 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.815951 11298 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:50.833838 9903 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:50.834210 11298 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.822708 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.836373 11298 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.836916 11343 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:50.837656 10560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54754 (request call id 215) took 1002 ms. Trace:
I20260430 07:58:50.838042 10560 rpcz_store.cc:276] 0430 07:58:49.835429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.836625 (+ 1196us) service_pool.cc:225] Handling call
0430 07:58:49.897497 (+ 60872us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:50.337557 (+440060us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.837418 (+499861us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54005c1a54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:50.837599 (+ 181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":828127488}
I20260430 07:58:50.840348 11323 consensus_queue.cc:1059] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260430 07:58:50.840617 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.841379 9937 raft_consensus.cc:2393] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:50.841486 11323 raft_consensus.cc:3055] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:50.841969 11323 raft_consensus.cc:740] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:50.842382 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.842501 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.842730 11323 consensus_queue.cc:260] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.843529 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.843938 11323 raft_consensus.cc:3060] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:50.844726 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.843300 9937 raft_consensus.cc:2410] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:58:50.845157 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.845731 10011 raft_consensus.cc:2376] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2. Re-sending same reply.
I20260430 07:58:50.845894 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.845973 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.846838 9934 raft_consensus.cc:2393] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:50.846949 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.847617 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.847976 10011 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:50.847930 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.848768 9937 raft_consensus.cc:2368] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:50.814213 11122 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:50.850867 11469 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:50.851799 11122 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:50.852444 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:50.853243 10560 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:50.765568 11423 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260430 07:58:50.822781 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.843626 11285 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:50.855720 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.897367 10565 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:58:50.892393 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:50.874495 9977 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:50.914057 9934 raft_consensus.cc:2393] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:50.923413 10011 raft_consensus.cc:1240] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:50.924034 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:50.926169 10003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1415) took 1813 ms. Trace:
I20260430 07:58:50.927953 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:50.928221 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:50.928790 9863 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:50.931188 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 96) took 1521 ms. Trace:
I20260430 07:58:50.931774 9858 rpcz_store.cc:276] 0430 07:58:49.410746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.452887 (+1042141us) service_pool.cc:225] Handling call
0430 07:58:50.769696 (+316809us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:50.772880 (+ 3184us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.930906 (+158026us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b540010cf50. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:50.931108 (+ 202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":726207744}
I20260430 07:58:50.926594 10003 rpcz_store.cc:276] 0430 07:58:49.112339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.579019 (+466680us) service_pool.cc:225] Handling call
0430 07:58:49.579488 (+ 469us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:50.283188 (+703700us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.925933 (+642745us) spinlock_profiling.cc:243] Waited 669 ms on lock 0x7b54003c0854. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:50.926115 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1539629952}
W20260430 07:58:50.938906 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:50.829169 11242 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:50.961714 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.964567 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:50.942866 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.965833 10003 raft_consensus.cc:2393] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:50.966017 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:50.966086 9930 raft_consensus.cc:1275] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:58:50.967666 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:50.771678 10928 consensus_queue.cc:237] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.971112 9977 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:50.973532 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.973994 9936 raft_consensus.cc:1240] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:50.974879 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:50.975241 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.976181 9936 raft_consensus.cc:2393] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:50.978732 10561 raft_consensus.cc:1240] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260430 07:58:50.979617 11323 consensus_queue.cc:1059] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:50.980489 11323 consensus_queue.cc:1048] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:50.980626 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.982159 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.983112 11480 consensus_queue.cc:1059] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:50.943635 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.983501 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.984149 11480 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:50.984548 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1159) took 1061 ms. Trace:
I20260430 07:58:50.984476 11023 consensus_queue.cc:1059] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:50.926198 11414 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:50.985060 9862 rpcz_store.cc:276] 0430 07:58:49.923605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.518426 (+594821us) service_pool.cc:225] Handling call
0430 07:58:50.518837 (+ 411us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:50.579954 (+ 61117us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:50.579997 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:50.635298 (+ 55301us) log.cc:844] Serialized 26 byte log entry
0430 07:58:50.758355 (+123057us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:50.758400 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:50.758463 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:50.984335 (+225872us) raft_consensus.cc:1685] finished
0430 07:58:50.984354 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:50.984492 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":534,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":56102,"thread_start_us":70378,"threads_started":1,"wal-append.queue_time_us":274512}
I20260430 07:58:50.987591 11414 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:50.988301 10559 raft_consensus.cc:1240] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:50.954316 9935 raft_consensus.cc:1240] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:50.954476 11469 consensus_queue.cc:1048] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:50.955850 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:50.954396 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:50.913913 11270 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.044s
I20260430 07:58:50.991206 11023 consensus_queue.cc:1059] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:58:50.968523 9930 raft_consensus.cc:2393] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:50.955155 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.910096 11337 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:50.926946 11122 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:58:50.995965 9858 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260430 07:58:50.995340 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:51.003296 10565 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:50.977056 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.996793 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:50.997484 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:50.995479 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.005622 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.007567 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:50.984513 11480 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:50.996006 11469 raft_consensus.cc:3055] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:50.997438 11023 raft_consensus.cc:3055] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:51.025864 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:37818 (request call id 228) took 1404 ms. Trace:
I20260430 07:58:50.998791 11270 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:51.008522 11468 consensus_queue.cc:1243] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer d66db415f0f0402fa431349afbf59888 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:58:50.999058 11343 raft_consensus.cc:3055] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:58:50.993913 11471 raft_consensus.cc:3055] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:51.018867 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.021027 11480 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:51.017921 9903 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:50.968526 11385 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.027249 9858 rpcz_store.cc:276] 0430 07:58:49.621084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:51.002905 (+1381821us) service_pool.cc:225] Handling call
0430 07:58:51.025222 (+ 22317us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:51.025923 11469 raft_consensus.cc:740] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:51.026577 11023 raft_consensus.cc:740] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:51.042290 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.040069 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:51.043319 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.011474 11298 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260430 07:58:51.035915 11471 raft_consensus.cc:740] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:51.051388 9932 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:51.051324 11471 consensus_queue.cc:260] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.052589 11471 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:51.035152 11343 raft_consensus.cc:740] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:51.046572 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:51.052477 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.035873 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:51.055160 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 433027us
I20260430 07:58:51.053439 11310 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:51.025717 11239 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:51.043845 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.071805 9935 raft_consensus.cc:2393] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.079118 9978 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:51.043530 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:51.083307 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.035012 11270 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:51.084985 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.084918 9863 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:51.099188 11270 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:51.100039 11023 consensus_queue.cc:260] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.101763 11023 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:51.102224 9930 raft_consensus.cc:2393] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.103525 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.105521 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1158) took 1182 ms. Trace:
I20260430 07:58:51.105911 9861 rpcz_store.cc:276] 0430 07:58:49.923248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.508126 (+584878us) service_pool.cc:225] Handling call
0430 07:58:50.569962 (+ 61836us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:50.619539 (+ 49577us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:50.619581 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:50.625192 (+ 5611us) log.cc:844] Serialized 26 byte log entry
0430 07:58:51.041670 (+416478us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:51.041717 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:51.041791 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:51.105260 (+ 63469us) raft_consensus.cc:1685] finished
0430 07:58:51.105324 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:51.105453 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9009,"raft.queue_time_us":49753,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":19133,"thread_start_us":368593,"threads_started":1,"wal-append.queue_time_us":458222}
I20260430 07:58:51.106963 11239 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.107986 10567 raft_consensus.cc:1240] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:51.109474 11469 consensus_queue.cc:260] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.110971 11379 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.113857 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1155) took 1401 ms. Trace:
I20260430 07:58:51.114321 9938 rpcz_store.cc:276] 0430 07:58:49.712855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.911995 (+199140us) service_pool.cc:225] Handling call
0430 07:58:49.912483 (+ 488us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:50.376490 (+464007us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:51.113422 (+736932us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54001ac554. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:51.113783 (+ 361us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1047836416}
I20260430 07:58:51.116231 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.116830 9863 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:58:51.074260 11440 consensus_meta.cc:287] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.641s user 0.008s sys 0.004s
I20260430 07:58:51.118846 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.118976 9978 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:51.119513 9935 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 1.
I20260430 07:58:51.121285 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:51.122278 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:51.122879 10006 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:51.130575 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.132119 9935 raft_consensus.cc:2393] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.140250 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.141328 9938 raft_consensus.cc:2410] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:58:51.141613 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.142102 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.142671 9938 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:51.142864 9935 raft_consensus.cc:1240] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:51.109261 10560 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:51.145182 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:51.147655 11479 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:51.148227 11479 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:51.148748 11479 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:51.146345 10533 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:51.150696 11414 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:51.151154 11414 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:51.151667 11414 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:51.157323 10566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54754 (request call id 204) took 2172 ms. Trace:
I20260430 07:58:51.162401 9938 raft_consensus.cc:1240] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 1. Current term is 2. Ops: []
W20260430 07:58:51.199101 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 197451us
I20260430 07:58:51.087512 9931 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:51.200963 9863 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:58:51.132820 9932 raft_consensus.cc:2393] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.202811 10534 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:51.204878 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:51.201331 11424 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.589s user 0.040s sys 0.046s
W20260430 07:58:51.206629 11424 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.594s user 0.040s sys 0.046s
W20260430 07:58:51.207475 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 138425us
I20260430 07:58:51.208918 9903 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:51.105219 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.132434 11323 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:51.228107 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1144) took 1617 ms. Trace:
I20260430 07:58:51.228538 9933 rpcz_store.cc:276] 0430 07:58:49.611030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:49.611224 (+ 194us) service_pool.cc:225] Handling call
0430 07:58:49.611514 (+ 290us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:49.633694 (+ 22180us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:51.227865 (+1594171us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b540036bfd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:51.228048 (+ 183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48764800}
I20260430 07:58:51.190764 11050 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:51.228565 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.101011 11343 consensus_queue.cc:260] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.121778 11440 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.193674 10566 rpcz_store.cc:276] 0430 07:58:48.985089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:48.988075 (+ 2986us) service_pool.cc:225] Handling call
0430 07:58:48.988467 (+ 392us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:49.623073 (+634606us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:51.156998 (+1533925us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b540060b3d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:51.157198 (+ 200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1239298816}
W20260430 07:58:51.256306 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.208165 11443 consensus_queue.cc:1059] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20260430 07:58:51.283385 11440 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.210381 11298 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:51.277038 11122 consensus_queue.cc:1048] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:51.278640 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.278833 11474 consensus_queue.cc:1048] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:51.279197 11338 consensus_queue.cc:1059] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:51.254123 11050 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.207218 11242 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:51.270485 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 43506.7us
I20260430 07:58:51.200340 11270 consensus_queue.cc:1048] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260430 07:58:51.228263 11310 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:51.292301 11379 consensus_queue.cc:1059] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260430 07:58:51.224046 11424 raft_consensus.cc:697] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 3 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:51.315838 11424 consensus_queue.cc:237] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.317129 9860 raft_consensus.cc:1240] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:51.277029 11385 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.317885 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1172) took 1356 ms. Trace:
I20260430 07:58:51.318225 9860 rpcz_store.cc:276] 0430 07:58:49.961834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.794261 (+832427us) service_pool.cc:225] Handling call
0430 07:58:51.316710 (+522449us) spinlock_profiling.cc:243] Waited 6.7 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:51.316954 (+ 244us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:51.317607 (+ 653us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:51.317829 (+ 222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15419776}
I20260430 07:58:51.320426 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1173) took 1334 ms. Trace:
I20260430 07:58:51.286989 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.321000 9857 rpcz_store.cc:276] 0430 07:58:49.986531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.796862 (+810331us) service_pool.cc:225] Handling call
0430 07:58:51.319613 (+522751us) spinlock_profiling.cc:243] Waited 6.48 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:58:51.319842 (+ 229us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:51.320030 (+ 188us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:51.320056 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:51.320132 (+ 76us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:51.320166 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:51.320230 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:51.320368 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14896896}
I20260430 07:58:51.323369 9932 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:51.349335 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:51.350425 9932 raft_consensus.cc:2393] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.351917 11484 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:51.352406 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.299109 9930 raft_consensus.cc:2393] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.304993 9931 raft_consensus.cc:2393] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.353526 11440 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:51.254923 11476 consensus_queue.cc:1059] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260430 07:58:51.354249 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.354118 11440 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.354595 11476 raft_consensus.cc:3055] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:58:51.355000 11476 raft_consensus.cc:740] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:51.355532 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.355767 11476 consensus_queue.cc:260] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.356665 9930 raft_consensus.cc:2410] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260430 07:58:51.310482 11483 log.cc:927] Time spent T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260430 07:58:51.310635 11242 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.357693 9932 raft_consensus.cc:2393] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.358541 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.311182 11493 raft_consensus.cc:3055] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:51.358989 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:51.359156 11493 raft_consensus.cc:740] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:51.313187 11050 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.325687 11298 raft_consensus.cc:697] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:51.321233 11479 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.310354 11404 consensus_queue.cc:1059] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:58:51.361191 11298 consensus_queue.cc:237] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:51.361874 11404 raft_consensus.cc:3055] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:58:51.362277 11404 raft_consensus.cc:740] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:51.362433 11479 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:51.363099 11404 consensus_queue.cc:260] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:51.360102 11493 consensus_queue.cc:260] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.365419 11404 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:51.326537 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1338) took 1167 ms. Trace:
I20260430 07:58:51.367252 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.330305 11414 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:51.357177 11476 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:51.356365 11440 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.313596 11310 raft_consensus.cc:697] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:51.332419 11480 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.370621 11310 consensus_queue.cc:237] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.371414 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.320461 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1168) took 1418 ms. Trace:
I20260430 07:58:51.365201 11493 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:51.231324 11323 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:51.352530 11484 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:51.370023 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:51.374006 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.415716 9938 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:51.417834 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.419176 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.419549 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:51.420765 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.438963 9861 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:51.442939 11414 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.447885 11343 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:51.366338 9856 rpcz_store.cc:276] 0430 07:58:50.158727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.853094 (+694367us) service_pool.cc:225] Handling call
0430 07:58:50.925255 (+ 72161us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:50.961655 (+ 36400us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:50.961693 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:50.962280 (+ 587us) log.cc:844] Serialized 26 byte log entry
0430 07:58:51.224885 (+262605us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:51.232839 (+ 7954us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:51.248518 (+ 15679us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:51.326249 (+ 77731us) raft_consensus.cc:1685] finished
0430 07:58:51.326285 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:51.326448 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13424,"raft.queue_time_us":20219,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"thread_start_us":240385,"threads_started":1,"wal-append.queue_time_us":356210}
I20260430 07:58:51.450443 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.452594 9856 raft_consensus.cc:1240] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:51.453822 11480 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:51.424804 10012 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.083s sys 0.092s
W20260430 07:58:51.455406 10012 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.993s user 0.084s sys 0.093s
I20260430 07:58:51.456212 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 95) took 1057 ms. Trace:
I20260430 07:58:51.456568 10012 rpcz_store.cc:276] 0430 07:58:50.398622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.400587 (+ 1965us) service_pool.cc:225] Handling call
0430 07:58:50.401046 (+ 459us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:50.461497 (+ 60451us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:50.461528 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:50.461974 (+ 446us) log.cc:844] Serialized 26 byte log entry
0430 07:58:51.455839 (+993865us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:58:51.455886 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:51.455965 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:51.456011 (+ 46us) raft_consensus.cc:1685] finished
0430 07:58:51.456022 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:51.456154 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20255,"raft.queue_time_us":34366,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"thread_start_us":1033189,"threads_started":2,"wal-append.queue_time_us":675361}
W20260430 07:58:51.427443 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:51.330968 11487 log.cc:927] Time spent T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.051s user 0.003s sys 0.000s
I20260430 07:58:51.374073 9929 rpcz_store.cc:276] 0430 07:58:49.901702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.240187 (+338485us) service_pool.cc:225] Handling call
0430 07:58:50.240494 (+ 307us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:50.437838 (+197344us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:51.320178 (+882340us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5400347954. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:51.320398 (+ 220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":405034752}
I20260430 07:58:51.460812 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.462183 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.462889 9929 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:51.461117 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.359021 11242 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.334728 11342 raft_consensus.cc:3055] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:51.464501 11342 raft_consensus.cc:740] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:51.464890 9937 raft_consensus.cc:1275] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:51.436442 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.414487 11323 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:51.438283 9931 raft_consensus.cc:2393] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.454241 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:51.463047 11472 log.cc:927] Time spent T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.706s user 0.001s sys 0.000s
I20260430 07:58:51.341012 11093 raft_consensus.cc:2955] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:51.463887 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.421202 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.366302 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.338197 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:51.510070 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.511103 9860 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:51.512732 9930 raft_consensus.cc:2393] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.465440 11342 consensus_queue.cc:260] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:51.440909 11488 log.cc:927] Time spent T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.160s user 0.000s sys 0.001s
I20260430 07:58:51.375046 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:51.529619 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 154584us
I20260430 07:58:51.382735 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:51.532442 11489 log.cc:927] Time spent T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.251s user 0.001s sys 0.001s
W20260430 07:58:51.477600 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.533663 10567 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:51.478133 11242 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:51.535907 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1340) took 1356 ms. Trace:
I20260430 07:58:51.535825 11242 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.536525 9854 rpcz_store.cc:276] 0430 07:58:50.180247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.941797 (+761550us) service_pool.cc:225] Handling call
0430 07:58:50.981631 (+ 39834us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:50.982018 (+ 387us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:50.982053 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:50.989592 (+ 7539us) log.cc:844] Serialized 26 byte log entry
0430 07:58:51.194398 (+204806us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:51.253418 (+ 59020us) log.cc:844] Serialized 14 byte log entry
0430 07:58:51.300341 (+ 46923us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:51.332322 (+ 31981us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:51.438635 (+106313us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b1800047c80. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb566159e1a 00007fb56f7f25c9 00007fb56d1653ef 00007fb5661e5bb1 00007fb5661e7e01 00007fb5661e7db9 00007fb5661e7d49 00007fb5661e7d11 00007fb5661e700d
0430 07:58:51.494084 (+ 55449us) spinlock_profiling.cc:243] Waited 6.39 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:58:51.535654 (+ 41570us) raft_consensus.cc:1685] finished
0430 07:58:51.535687 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:51.535846 (+ 159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34433,"prepare.queue_time_us":31129,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":10117,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":232,"spinlock_wait_cycles":23511680,"thread_start_us":84427,"threads_started":1,"wal-append.queue_time_us":246351}
I20260430 07:58:51.538797 11242 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.480821 9933 raft_consensus.cc:2393] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:51.540220 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:51.490625 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.545118 9904 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:51.548744 9904 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:51.517820 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.520982 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:51.549561 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:51.532526 10928 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:58:51.535148 9828 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:51.550992 9938 raft_consensus.cc:2393] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:51.552434 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:51.553109 9862 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:51.553454 9828 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:51.554075 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:51.554663 9862 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:51.555217 9828 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:51.555474 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.556062 9862 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:51.557740 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:51.467761 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:51.558780 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:51.562536 10004 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:51.324342 9759 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53852 (request call id 90) took 7236 ms. Trace:
I20260430 07:58:51.528025 9929 raft_consensus.cc:2368] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3.
I20260430 07:58:51.579144 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:51.581537 9904 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:51.582957 9861 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:51.583181 9904 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:51.564663 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.488498 9761 catalog_manager.cc:5671] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
W20260430 07:58:51.584832 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.576508 9863 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260430 07:58:51.588644 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:51.592453 9904 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:51.593146 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.593665 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:51.600314 10003 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:51.533579 11239 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:51.603588 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:51.605201 9863 raft_consensus.cc:1275] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260430 07:58:51.607352 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:51.608359 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.563751 9759 rpcz_store.cc:276] 0430 07:58:44.087614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.087817 (+ 203us) service_pool.cc:225] Handling call
0430 07:58:44.252760 (+164943us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b540001fc54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ed470 00007fb56928c31a 00007fb56f02f46c 00007fb56f1293ee 00007fb56d60f4a4 00007fb56d60f432 00007fb56d60f361 00007fb56d60f2dc 00007fb56d60e592 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:51.324278 (+7071518us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:58:48.486189 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:58:48.613662 (+127473us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:58:48.613706 (+ 44us) write_op.cc:435] Acquired schema lock
0430 07:58:48.613724 (+ 18us) tablet.cc:662] Decoding operations
0430 07:58:48.645292 (+ 31568us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:58:48.648521 (+ 3229us) write_op.cc:641] Partition lock acquired for write op
0430 07:58:48.684642 (+ 36121us) tablet.cc:685] Acquiring locks for 5 operations
0430 07:58:48.893106 (+208464us) tablet.cc:701] Row locks acquired
0430 07:58:48.893135 (+ 29us) write_op.cc:260] PREPARE: finished
0430 07:58:48.944341 (+ 51206us) write_op.cc:270] Start()
0430 07:58:48.986114 (+ 41773us) write_op.cc:276] Timestamp: P: 1777535928926758 usec, L: 0
0430 07:58:49.007207 (+ 21093us) op_driver.cc:348] REPLICATION: starting
0430 07:58:49.082453 (+ 75246us) log.cc:844] Serialized 2131 byte log entry
0430 07:58:49.850203 (+767750us) op_driver.cc:464] REPLICATION: finished
0430 07:58:50.133931 (+283728us) write_op.cc:301] APPLY: starting
0430 07:58:50.134032 (+ 101us) tablet.cc:1367] starting BulkCheckPresence
0430 07:58:50.134207 (+ 175us) tablet.cc:1370] finished BulkCheckPresence
0430 07:58:50.134343 (+ 136us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:58:50.158212 (+ 23869us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:58:50.184405 (+ 26193us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0430 07:58:50.188034 (+ 3629us) write_op.cc:312] APPLY: finished
0430 07:58:50.211997 (+ 23963us) log.cc:844] Serialized 46 byte log entry
0430 07:58:50.212220 (+ 223us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:50.212340 (+ 120us) write_op.cc:454] Released schema lock
0430 07:58:50.212386 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0430 07:58:50.212739 (+ 353us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:50.212767 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":93621,"reported_tablets":9,"spinlock_wait_cycles":88381312,"child_traces":[["op",{"apply.queue_time_us":242593,"apply.run_cpu_time_us":3809,"apply.run_wall_time_us":78809,"num_ops":5,"prepare.queue_time_us":2102101,"prepare.run_cpu_time_us":39705,"prepare.run_wall_time_us":729324,"raft.queue_time_us":291695,"raft.run_cpu_time_us":11971,"raft.run_wall_time_us":279588,"replication_time_us":790276,"thread_start_us":2349710,"threads_started":4,"wal-append.queue_time_us":347031,"wal-append.run_cpu_time_us":23525,"wal-append.run_wall_time_us":1799103}]]}
W20260430 07:58:51.613543 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.616403 9904 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:51.623754 9904 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:51.602666 11494 raft_consensus.cc:2804] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:51.586973 11403 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:51.589970 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.589229 11285 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:51.600186 11242 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:51.588817 11342 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:51.602855 11400 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:51.599545 11050 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260430 07:58:51.627133 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:51.667790 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 405827us
I20260430 07:58:51.674618 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:51.676519 9862 raft_consensus.cc:2410] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:58:51.677031 10012 raft_consensus.cc:1240] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:51.679546 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:51.588414 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:51.610203 11383 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260430 07:58:51.552822 11413 consensus_queue.cc:1059] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260430 07:58:51.640831 9937 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:51.642071 10567 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:51.642894 11484 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:51.643954 9860 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:51.644161 11404 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:51.622627 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.593851 9761 catalog_manager.cc:5671] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:51.589542 11440 consensus_queue.cc:1059] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260430 07:58:51.673797 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:51.656862 11494 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:51.619786 11239 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:51.715852 9978 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:51.718493 11469 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:58:51.723389 11487 log.cc:927] Time spent T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260430 07:58:51.723759 11404 raft_consensus.cc:697] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:51.726820 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1345) took 1480 ms. Trace:
I20260430 07:58:51.753552 9859 rpcz_store.cc:276] 0430 07:58:50.246524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.447102 (+200578us) service_pool.cc:225] Handling call
0430 07:58:51.726575 (+1279473us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540029e354. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:51.726753 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2843808128}
I20260430 07:58:51.622285 11414 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
W20260430 07:58:51.766762 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.731410 9761 catalog_manager.cc:5671] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:51.768038 9761 catalog_manager.cc:5671] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 3, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:51.733176 10004 raft_consensus.cc:2410] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260430 07:58:51.732817 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 470682us
I20260430 07:58:51.695613 11403 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:51.686467 11400 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:51.746376 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 167233us
I20260430 07:58:51.715073 11285 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:51.772823 9828 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:51.776923 9761 catalog_manager.cc:5671] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:51.779615 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:51.772846 9977 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:58:51.775539 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:51.746474 11379 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:51.779997 9761 catalog_manager.cc:5671] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:51.744163 11494 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:51.793000 11379 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:51.793852 11497 log.cc:927] Time spent T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260430 07:58:51.795289 11379 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.796351 11424 raft_consensus.cc:3055] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:51.796785 11424 raft_consensus.cc:740] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:51.797746 11502 raft_consensus.cc:2749] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:51.798530 11310 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:51.799003 11310 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.799517 11242 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:58:51.783164 9977 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:51.782953 9761 catalog_manager.cc:5671] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:51.749287 11443 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:51.830767 9828 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:51.840143 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 56993.2us
I20260430 07:58:51.839941 9828 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:51.747750 11496 consensus_queue.cc:1059] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260430 07:58:51.824990 11484 raft_consensus.cc:3055] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:51.800899 11424 consensus_queue.cc:260] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.782465 11285 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:51.837826 9761 catalog_manager.cc:5671] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:51.778170 11400 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:51.785560 9854 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:51.785729 11440 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:51.800984 11310 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.804358 11323 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.842357 9977 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:51.806524 11385 raft_consensus.cc:3055] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:51.855259 9828 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:51.841473 10003 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
W20260430 07:58:51.860569 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 115015us
W20260430 07:58:51.881305 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 257480us
I20260430 07:58:51.863801 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:51.885767 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:51.878821 11298 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.006s sys 0.019s
I20260430 07:58:51.915021 9903 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:51.882418 9977 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:51.871032 9761 catalog_manager.cc:5671] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:51.916229 9904 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:51.923555 9863 raft_consensus.cc:1240] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:51.925631 9862 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260430 07:58:51.925982 9860 raft_consensus.cc:1240] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:51.857604 11285 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:51.843796 11413 raft_consensus.cc:2804] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:51.960258 9761 catalog_manager.cc:5671] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 3, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:51.863322 11239 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:51.855527 11424 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:51.959797 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 77434.4us
I20260430 07:58:51.958832 9903 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:58:51.878226 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 346631us
I20260430 07:58:51.847635 11443 raft_consensus.cc:697] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:51.972427 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 56286.8us
W20260430 07:58:51.941675 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 87410.7us
I20260430 07:58:51.993372 9828 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:51.960260 11403 raft_consensus.cc:2764] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260430 07:58:51.844018 11242 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:51.861493 11404 consensus_queue.cc:237] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:51.974931 9761 catalog_manager.cc:5671] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:51.975625 11480 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:51.854282 11484 raft_consensus.cc:740] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:58:51.849962 11342 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.111s
I20260430 07:58:51.893695 11476 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:51.854028 11496 raft_consensus.cc:3055] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
W20260430 07:58:51.960888 11298 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s user 0.008s sys 0.026s
I20260430 07:58:51.915886 11418 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:51.970992 11379 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:51.992286 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.972968 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:52.010187 11480 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:58:51.938380 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:52.032363 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:52.032469 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.972555 11413 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:52.000872 9761 catalog_manager.cc:5671] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
I20260430 07:58:52.015749 11511 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:51.996610 11383 consensus_queue.cc:1059] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260430 07:58:51.916627 11502 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:51.978101 11092 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:51.992835 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:51.992061 11499 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:52.029891 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:51.995323 9904 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:52.032109 11480 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:52.021493 11050 consensus_queue.cc:1059] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260430 07:58:52.037125 9761 catalog_manager.cc:5671] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 3 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
I20260430 07:58:51.909034 11385 raft_consensus.cc:740] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:51.994110 11491 consensus_queue.cc:1059] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260430 07:58:51.971191 11493 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:52.022387 11496 raft_consensus.cc:740] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:52.072762 11496 consensus_queue.cc:260] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.074927 11496 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:52.033231 9935 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:52.077497 11510 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:52.077958 11510 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:52.078308 11510 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:52.037371 11508 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:52.078845 11499 raft_consensus.cc:697] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:52.079916 11499 consensus_queue.cc:237] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.080299 11493 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.081682 11050 raft_consensus.cc:3055] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:52.082089 11050 raft_consensus.cc:740] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:52.082406 11493 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:52.083472 11503 log.cc:927] Time spent T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260430 07:58:52.082926 11050 consensus_queue.cc:260] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.084405 11050 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:52.060921 9761 catalog_manager.cc:5671] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:52.086354 11491 raft_consensus.cc:3055] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:52.087023 11491 raft_consensus.cc:740] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:52.040346 11413 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:52.062281 10011 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
W20260430 07:58:52.062958 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 67835.7us
I20260430 07:58:52.092803 9904 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:52.052224 11092 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:52.061185 9863 raft_consensus.cc:1240] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:52.076670 11494 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.119150 11494 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:52.090492 9761 catalog_manager.cc:5671] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:52.062148 11443 consensus_queue.cc:237] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.106464 9904 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:52.107097 11092 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:52.108922 11383 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:58:52.090505 11504 log.cc:927] Time spent T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260430 07:58:52.109513 11323 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:52.022212 11484 consensus_queue.cc:260] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.065048 11506 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:52.109891 11476 raft_consensus.cc:3055] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:52.044216 11511 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.034976 11418 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.124516 9761 catalog_manager.cc:5671] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
I20260430 07:58:52.179097 9761 catalog_manager.cc:5671] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 3, leader changed from d66db415f0f0402fa431349afbf59888 (127.8.119.132) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 3 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:52.143357 9904 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:52.184928 9904 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:52.188333 9904 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:52.181236 9761 catalog_manager.cc:5671] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:52.190853 9761 catalog_manager.cc:5671] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:52.173034 11494 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:52.172485 11383 raft_consensus.cc:697] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:52.213614 11494 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:52.214115 11494 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:52.214264 11383 consensus_queue.cc:237] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.216540 11413 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:52.218166 9858 raft_consensus.cc:2393] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 4: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:52.173648 11341 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:52.220923 9858 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260430 07:58:52.226145 9856 raft_consensus.cc:1240] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [1.3-3.4]
I20260430 07:58:52.227106 11310 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:52.201169 11379 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:52.227833 11310 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:52.203377 11511 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:52.177361 11506 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.228508 9761 catalog_manager.cc:5671] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:52.230535 11506 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:52.231559 11242 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:52.232362 11242 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:52.235890 11242 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:52.088707 11491 consensus_queue.cc:260] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.238040 11491 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:52.240739 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:52.171600 11476 raft_consensus.cc:740] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:52.245633 11341 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:58:52.250268 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 269918us
I20260430 07:58:52.266178 9977 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:52.226405 11385 consensus_queue.cc:260] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.179679 11239 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:52.252485 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 259090us
I20260430 07:58:52.280843 9828 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:52.225416 11494 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.286710 9978 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:52.231773 9761 catalog_manager.cc:5671] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:58:52.169570 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1165) took 2243 ms. Trace:
W20260430 07:58:52.319270 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:52.320200 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:52.322518 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:52.323241 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:52.333635 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:52.334112 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:52.334627 10006 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:52.335124 10003 raft_consensus.cc:1240] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:52.340022 10012 raft_consensus.cc:1275] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:52.347499 10533 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:52.290268 11469 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:58:52.244649 11484 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:52.283700 11242 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:52.292814 11418 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:52.283668 11323 consensus_queue.cc:1059] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.011s
W20260430 07:58:52.295639 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:52.300354 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:52.285472 9854 raft_consensus.cc:2410] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260430 07:58:52.305461 11476 consensus_queue.cc:260] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.285903 11239 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.279940 11501 consensus_queue.cc:1059] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260430 07:58:52.281597 11502 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:52.354105 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 64953.7us
I20260430 07:58:52.355633 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:52.413959 9934 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:52.355909 10011 raft_consensus.cc:1275] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260430 07:58:52.415020 10011 pending_rounds.cc:85] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 2
I20260430 07:58:52.415547 10011 pending_rounds.cc:107] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260430 07:58:52.308779 11480 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:52.356927 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:52.431013 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 242650us
I20260430 07:58:52.275239 9977 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:52.362553 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:52.363384 10012 raft_consensus.cc:1275] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:58:52.317598 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:52.365818 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:52.434685 10567 raft_consensus.cc:2393] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:52.326248 11341 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:52.387609 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:52.442431 10006 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:52.388839 9903 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:52.443727 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 54975.4us
I20260430 07:58:52.368680 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:52.354079 11385 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:52.353622 11512 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:52.363533 10928 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:52.415993 10011 raft_consensus.cc:2889] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:52.432057 9904 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:52.477852 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 45878.4us
I20260430 07:58:52.478856 9904 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:52.449440 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:52.404270 9857 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 1.
I20260430 07:58:52.480120 9904 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:52.446067 9903 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:52.484866 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:58:52.435600 11470 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:52.387001 11242 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.495915 9904 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:52.444597 11469 raft_consensus.cc:3055] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:52.369354 11525 consensus_queue.cc:1059] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
I20260430 07:58:52.472487 11404 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260430 07:58:52.514756 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 73420.1us
I20260430 07:58:52.508658 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:52.466045 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:52.525364 10006 raft_consensus.cc:2410] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260430 07:58:52.525396 10534 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:58:52.528806 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 253509us
I20260430 07:58:52.510310 10012 raft_consensus.cc:1275] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260430 07:58:52.529172 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:52.506623 11383 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:52.507795 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:52.472859 11394 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:52.480633 11493 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:52.531790 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:52.472502 11379 consensus_queue.cc:1059] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260430 07:58:52.449311 11092 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.472641 11310 consensus_queue.cc:1048] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260430 07:58:52.480969 11093 consensus_queue.cc:1059] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20260430 07:58:52.480937 11341 consensus_queue.cc:1059] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
W20260430 07:58:52.513880 11285 consensus_meta.cc:287] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.522s user 0.011s sys 0.000s
I20260430 07:58:52.472465 11511 consensus_queue.cc:1048] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260430 07:58:52.577524 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:52.604051 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:52.514207 11413 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:52.440584 11400 consensus_meta.cc:287] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.567s user 0.013s sys 0.000s
I20260430 07:58:52.515534 10928 raft_consensus.cc:697] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
W20260430 07:58:52.589052 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 64347.1us
W20260430 07:58:52.589452 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 59852.3us
I20260430 07:58:52.490583 11519 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:52.609243 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 163162us
I20260430 07:58:52.513157 11469 raft_consensus.cc:740] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:52.607041 11336 raft_consensus.cc:3055] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:52.523281 11443 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:52.444007 11476 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:52.621019 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:52.520635 11323 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:52.530256 11499 raft_consensus.cc:2804] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:52.512116 11512 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.515967 11470 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.588557 11521 consensus_queue.cc:1059] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.232s
I20260430 07:58:52.524127 11510 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:52.605432 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 214201us
I20260430 07:58:52.559183 9856 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:52.315263 9855 rpcz_store.cc:276] 0430 07:58:49.925746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.747948 (+822202us) service_pool.cc:225] Handling call
0430 07:58:50.811763 (+ 63815us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:58:50.941580 (+129817us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:50.971121 (+ 29541us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:58:50.971823 (+ 702us) log.cc:844] Serialized 52 byte log entry
0430 07:58:51.213450 (+241627us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:51.313498 (+100048us) spinlock_profiling.cc:243] Waited 9.14 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:58:51.627922 (+314424us) log.cc:844] Serialized 14 byte log entry
0430 07:58:52.153127 (+525205us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:52.168969 (+ 15842us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:52.169067 (+ 98us) raft_consensus.cc:1685] finished
0430 07:58:52.169210 (+ 143us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:52.169477 (+ 267us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":205745,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":63298,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":56140,"spinlock_wait_cycles":21030272,"thread_start_us":1022238,"threads_started":3,"wal-append.queue_time_us":308111,"wal-append.run_cpu_time_us":22299,"wal-append.run_wall_time_us":1530125}
W20260430 07:58:53.401260 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:53.414973 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:53.399048 11552 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:53.417114 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1354) took 1938 ms. Trace:
I20260430 07:58:53.418684 9854 rpcz_store.cc:276] 0430 07:58:51.478640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:51.577373 (+ 98733us) service_pool.cc:225] Handling call
0430 07:58:53.416474 (+1839101us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b54001cd4d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:53.416871 (+ 397us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97336064}
I20260430 07:58:53.421130 11552 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.402307 11551 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:53.426388 11551 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.429471 11551 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.430243 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1198) took 1446 ms. Trace:
I20260430 07:58:53.431066 11552 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:53.430928 9858 rpcz_store.cc:276] 0430 07:58:51.983599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:52.222794 (+239195us) service_pool.cc:225] Handling call
0430 07:58:52.223175 (+ 381us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:58:52.236935 (+ 13760us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:52.236965 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:58:52.244665 (+ 7700us) log.cc:844] Serialized 52 byte log entry
0430 07:58:52.444484 (+199819us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:52.493807 (+ 49323us) log.cc:844] Serialized 14 byte log entry
0430 07:58:52.494174 (+ 367us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:52.494350 (+ 176us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:52.878924 (+384574us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b1800063280. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661e5b62 00007fb5661e7e01 00007fb5661e7db9 00007fb5661e7d49 00007fb5661e7d11 00007fb5661e700d 0000000000379254 0000000000379099 00007fb5661c8fba
0430 07:58:53.429621 (+550697us) raft_consensus.cc:1685] finished
0430 07:58:53.429773 (+ 152us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:53.430014 (+ 241us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":49050,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":37451,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":28618,"spinlock_wait_cycles":31375232,"thread_start_us":160225,"threads_started":1,"wal-append.queue_time_us":544133}
I20260430 07:58:53.432174 9856 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:53.434717 11552 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.437410 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 127) took 1317 ms. Trace:
I20260430 07:58:53.437865 9859 rpcz_store.cc:276] 0430 07:58:52.120066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:52.616567 (+496501us) service_pool.cc:225] Handling call
0430 07:58:53.436330 (+819763us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:53.436826 (+ 496us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:53.436860 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:53.436955 (+ 95us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:53.436988 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:53.437060 (+ 72us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:53.437197 (+ 137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:53.436975 11552 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.440858 9861 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:53.440999 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 130) took 1131 ms. Trace:
I20260430 07:58:53.442742 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1188) took 1516 ms. Trace:
I20260430 07:58:53.441606 9863 rpcz_store.cc:276] 0430 07:58:52.309722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:52.840491 (+530769us) service_pool.cc:225] Handling call
0430 07:58:52.840836 (+ 345us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:52.841050 (+ 214us) raft_consensus.cc:1534] Early marking committed up to index 6
0430 07:58:53.064920 (+223870us) log.cc:844] Serialized 14 byte log entry
0430 07:58:53.065296 (+ 376us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:53.440235 (+374939us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:58:53.440417 (+ 182us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:53.440558 (+ 141us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:53.440803 (+ 245us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":37877,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":726,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":133,"thread_start_us":202522,"threads_started":1}
I20260430 07:58:53.443202 9861 rpcz_store.cc:276] 0430 07:58:51.926027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:51.926176 (+ 149us) service_pool.cc:225] Handling call
0430 07:58:53.440510 (+1514334us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:53.441424 (+ 914us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:53.442491 (+ 1067us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54001cd4d0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:53.442683 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3367182208}
W20260430 07:58:52.950848 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 454788us
I20260430 07:58:53.448354 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1207) took 1209 ms. Trace:
I20260430 07:58:53.449314 9856 rpcz_store.cc:276] 0430 07:58:52.238551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:52.264573 (+ 26022us) service_pool.cc:225] Handling call
0430 07:58:53.447797 (+1183224us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54001905d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:53.448230 (+ 433us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232663936}
W20260430 07:58:52.882781 11528 log.cc:927] Time spent T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.368s user 0.003s sys 0.000s
W20260430 07:58:53.080292 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 446589us
I20260430 07:58:52.865715 11239 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:52.607466 11493 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:53.479228 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:53.479791 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:53.479394 10928 consensus_queue.cc:237] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.887414 11418 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:52.864578 11508 consensus_meta.cc:287] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.786s user 0.010s sys 0.000s
I20260430 07:58:52.627794 11285 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.481563 11400 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:52.867848 11480 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:53.482285 9977 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:58:53.089694 11385 consensus_meta.cc:287] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.616s user 0.004s sys 0.008s
I20260430 07:58:53.484153 11336 raft_consensus.cc:740] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:53.484512 11400 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.481664 11418 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:52.601086 11414 raft_consensus.cc:2804] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260430 07:58:53.485821 11414 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:53.486299 11414 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 1 FOLLOWER]: Advancing to term 2
W20260430 07:58:52.841873 11522 log.cc:927] Time spent T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.328s user 0.000s sys 0.001s
I20260430 07:58:53.487073 11418 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:52.587548 11050 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:53.488538 11050 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:52.531248 11404 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:53.489179 11050 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:52.877384 11379 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:58:52.877211 11383 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:53.491176 11383 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:53.491533 11383 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:53.484200 11285 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.492950 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:52.867419 11424 consensus_meta.cc:287] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.874s user 0.009s sys 0.001s
I20260430 07:58:53.493731 9904 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:53.494318 9856 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:58:52.753650 11545 consensus_queue.cc:1048] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.741s
I20260430 07:58:52.618235 11394 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.496909 11499 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:53.497658 11499 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:53.498557 11508 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:53.500159 11336 consensus_queue.cc:260] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.449788 11552 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:52.823701 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:52.890985 11511 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:58:52.854803 11491 consensus_meta.cc:287] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.616s user 0.010s sys 0.001s
W20260430 07:58:53.052008 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 422751us
W20260430 07:58:53.157418 11544 log.cc:927] Time spent T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
W20260430 07:58:52.744854 11534 log.cc:927] Time spent T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
W20260430 07:58:53.028420 11502 consensus_meta.cc:287] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.590s user 0.010s sys 0.000s
W20260430 07:58:52.861035 11440 consensus_meta.cc:287] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.980s user 0.011s sys 0.000s
W20260430 07:58:52.744009 11533 log.cc:927] Time spent T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260430 07:58:53.605039 11323 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.605858 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1189) took 1679 ms. Trace:
I20260430 07:58:53.605736 11511 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:52.749981 11518 log.cc:927] Time spent T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20260430 07:58:53.502810 11404 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:58:52.924955 11484 consensus_meta.cc:287] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.559s user 0.007s sys 0.003s
W20260430 07:58:52.862282 11496 consensus_meta.cc:287] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.786s user 0.008s sys 0.002s
I20260430 07:58:53.598264 11336 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:52.859606 11092 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.633208 11496 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:53.636675 11480 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.634320 11400 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:53.637187 11496 raft_consensus.cc:697] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:53.634868 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 141510us
I20260430 07:58:53.637498 11480 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:53.637260 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:53.189393 11543 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.249190 11547 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:52.926304 11530 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:52.878227 11542 raft_consensus.cc:3055] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:52.628635 11494 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:52.617740 11403 raft_consensus.cc:3055] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:53.638850 11496 consensus_queue.cc:237] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.639482 9904 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:53.649534 11400 consensus_queue.cc:1059] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260430 07:58:53.651029 9828 connection.cc:254] RPC call timeout handler was delayed by -1.0908369140000218s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1758s for a response
I20260430 07:58:52.841742 11242 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:52.870716 11501 raft_consensus.cc:3055] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
W20260430 07:58:53.664862 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 182877us
I20260430 07:58:53.615760 11508 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.672418 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 106) took 1151 ms. Trace:
I20260430 07:58:53.668881 11323 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:53.679176 9828 connection.cc:254] RPC call timeout handler was delayed by -1.0621028769999725s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20260430 07:58:53.614204 9862 rpcz_store.cc:276] 0430 07:58:51.926725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:51.958740 (+ 32015us) service_pool.cc:225] Handling call
0430 07:58:51.959118 (+ 378us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:58:51.973334 (+ 14216us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:51.973379 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:58:51.974118 (+ 739us) log.cc:844] Serialized 52 byte log entry
0430 07:58:52.317720 (+343602us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:52.354460 (+ 36740us) log.cc:844] Serialized 14 byte log entry
0430 07:58:52.493063 (+138603us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:52.493133 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:53.597197 (+1104064us) raft_consensus.cc:1685] finished
0430 07:58:53.599956 (+ 2759us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:53.605609 (+ 5653us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":200,"prepare.queue_time_us":38203,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":84,"raft.queue_time_us":149281,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":35374,"thread_start_us":442581,"threads_started":2,"wal-append.queue_time_us":417273}
I20260430 07:58:52.617995 11520 consensus_queue.cc:1048] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 1.058s
I20260430 07:58:53.687443 11546 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.650918 11511 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:53.692910 9828 connection.cc:254] RPC call timeout handler was delayed by -1.0255264329999818s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1805s for a response
I20260430 07:58:53.684945 10009 rpcz_store.cc:276] 0430 07:58:52.520945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:52.530284 (+ 9339us) service_pool.cc:225] Handling call
0430 07:58:52.530557 (+ 273us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:53.632315 (+1101758us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:53.632423 (+ 108us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:53.632498 (+ 75us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:53.632573 (+ 75us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:53.671889 (+ 39316us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540018ced4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:53.671944 (+ 55us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:53.672194 (+ 250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2529910656}
W20260430 07:58:53.694609 11554 log.cc:927] Time spent T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.053s user 0.000s sys 0.002s
I20260430 07:58:53.697857 11512 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.698210 11511 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:58:53.699492 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:53.697956 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 58652.1us
I20260430 07:58:53.700214 11424 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.704449 11508 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.704944 11424 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.705082 11469 consensus_queue.cc:260] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.700259 11543 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.707278 11508 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.713744 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1205) took 1674 ms. Trace:
I20260430 07:58:53.699066 11519 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.718030 11394 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.719287 11547 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.725857 11542 raft_consensus.cc:740] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:53.726464 11530 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.727553 10533 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:53.728026 11502 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:53.728758 9904 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:53.732721 11499 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: NON_VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: true } }
I20260430 07:58:53.733696 11508 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:53.737884 11512 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:53.773613 11508 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.773701 11512 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.776057 11508 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.745292 11501 raft_consensus.cc:740] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:53.776973 11508 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.777957 11508 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.778541 11499 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.781599 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:53.782471 9860 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:53.782750 11520 consensus_queue.cc:1059] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260430 07:58:53.746316 11440 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:58:53.748116 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:53.719388 11546 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.754333 11510 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:53.717711 9828 leader_election.cc:336] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.744s (SENT)
I20260430 07:58:53.724852 11480 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:53.791077 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 101823us
I20260430 07:58:53.776899 11512 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.794844 9828 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:53.783803 11403 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:58:53.801115 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 83464us
I20260430 07:58:53.801007 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:53.801910 11239 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:53.802471 11512 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:58:53.804986 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 124909us
I20260430 07:58:53.804605 11414 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.806849 11546 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.783937 11424 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.808099 11508 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.795017 11394 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.808787 11547 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:53.809728 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:53.809947 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 81344.5us
I20260430 07:58:53.810369 11310 raft_consensus.cc:3055] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:53.810912 11499 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:53.813548 11470 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:53.813745 11519 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:53.815403 11510 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.724381 9857 rpcz_store.cc:276] 0430 07:58:52.005588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:52.542157 (+536569us) service_pool.cc:225] Handling call
0430 07:58:52.612768 (+ 70611us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:53.411424 (+798656us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:53.412724 (+ 1300us) log.cc:844] Serialized 14 byte log entry
0430 07:58:53.608405 (+195681us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:53.615474 (+ 7069us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:53.621009 (+ 5535us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:53.641116 (+ 20107us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:53.679460 (+ 38344us) inbound_call.cc:177] Queueing success response
0430 07:58:53.688544 (+ 9084us) spinlock_profiling.cc:243] Waited 41.3 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
Metrics: {"mutex_wait_us":18443,"prepare.queue_time_us":57202,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":8302,"raft.run_cpu_time_us":467,"raft.run_wall_time_us":69141,"spinlock_wait_cycles":94878464,"thread_start_us":91362,"threads_started":1,"wal-append.queue_time_us":222732}
I20260430 07:58:53.829336 11493 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:53.829582 10559 raft_consensus.cc:2393] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 4: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:53.720468 11558 consensus_queue.cc:1048] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.189s
I20260430 07:58:53.834484 10567 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260430 07:58:53.835009 11394 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.854594 11510 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.855950 9860 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:53.856416 11394 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.838152 9904 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:53.839589 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:53.806928 11530 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:53.849617 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:53.828372 11543 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.836676 11563 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.833474 11310 raft_consensus.cc:740] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:53.864611 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:53.865751 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:53.843150 11499 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:53.856986 11510 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:53.868692 11469 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:53.880714 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:53.881449 9904 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:58:53.880915 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:53.882381 11543 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:53.878785 11499 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:53.885342 11563 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.894455 11443 raft_consensus.cc:3055] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:53.894791 11443 raft_consensus.cc:740] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:53.895694 11443 consensus_queue.cc:260] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.891009 11418 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:53.896306 11501 consensus_queue.cc:260] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.892380 10011 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:53.896631 11443 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:53.897949 11501 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:53.898617 10011 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 909) took 1390 ms (client timeout 1540 ms). Trace:
W20260430 07:58:53.899089 10011 rpcz_store.cc:269] 0430 07:58:52.507717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:52.510375 (+ 2658us) service_pool.cc:225] Handling call
0430 07:58:53.898382 (+1388007us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54000cd354. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:58:53.898540 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2931275136}
I20260430 07:58:53.900137 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:53.846971 11542 consensus_queue.cc:260] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.889593 11543 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:53.902387 11542 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:53.902624 11385 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:53.903451 11385 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.904122 9904 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:53.904400 10534 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:53.905637 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:53.905907 11414 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.906214 11564 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:53.907007 11494 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:53.906666 9904 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:53.907620 11494 raft_consensus.cc:697] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:53.890983 11310 consensus_queue.cc:260] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.908954 11494 consensus_queue.cc:237] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.909819 11310 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:53.907441 11400 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:53.911175 11400 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.869355 11383 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.906693 11564 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:53.884698 11561 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.913867 11385 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.914014 11564 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:53.914407 11561 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:53.891625 11510 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:53.917747 9931 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561"
dest_uuid: "77e19ab653f540339c1c613fad1451b5"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:52704
I20260430 07:58:53.918118 11563 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:53.918707 9931 raft_consensus.cc:606] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
W20260430 07:58:53.913750 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 54796.7us
I20260430 07:58:53.920182 9977 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:53.921078 11563 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:53.922560 11403 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.902817 11543 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:53.924049 11403 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:53.924667 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:53.925832 9937 raft_consensus.cc:2410] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260430 07:58:53.925608 11530 consensus_queue.cc:1048] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:53.927268 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:53.928117 9929 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:53.929868 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:53.922570 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:53.930822 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:53.931063 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.931298 9935 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:53.931789 10012 raft_consensus.cc:2393] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:53.932125 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.933014 10011 raft_consensus.cc:2393] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:53.933087 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.935108 10008 raft_consensus.cc:1275] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260430 07:58:53.935510 10008 pending_rounds.cc:85] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 2
I20260430 07:58:53.935746 10008 pending_rounds.cc:107] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260430 07:58:53.936059 10008 raft_consensus.cc:2889] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:53.936510 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.937677 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:53.937841 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.937965 11385 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:53.938418 10559 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:53.939095 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.937201 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.938572 10004 raft_consensus.cc:1275] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:53.914988 11383 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.943442 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:53.943658 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.943948 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:53.937631 11511 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:53.944514 9857 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:53.944898 11511 raft_consensus.cc:697] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:53.947268 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:53.947849 11511 consensus_queue.cc:237] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:53.949512 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:53.950101 9934 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:53.945149 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:53.951107 9929 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:53.951215 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.946286 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:53.954048 9935 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:53.956184 11400 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:53.946732 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:53.956730 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 50174.8us
I20260430 07:58:53.958323 10010 raft_consensus.cc:2393] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:53.937675 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.958544 9904 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:53.959129 10012 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:53.944454 10007 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:53.953024 10011 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260430 07:58:53.964569 11499 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:53.965597 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 175522us
I20260430 07:58:53.969262 10559 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:53.969412 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:53.969995 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:53.970781 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:53.971213 11561 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:53.977814 10567 raft_consensus.cc:2393] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 4: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:53.979606 10566 raft_consensus.cc:2393] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
W20260430 07:58:53.982105 9828 leader_election.cc:336] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.502s (SENT)
I20260430 07:58:53.986712 9828 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:53.990425 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:53.990724 11499 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.993549 9929 raft_consensus.cc:1240] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [1.3-3.4]
I20260430 07:58:53.938974 11385 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.000461 11385 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:53.972366 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.003059 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:53.956852 11400 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
W20260430 07:58:54.005921 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.006958 9937 raft_consensus.cc:1240] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
W20260430 07:58:54.008946 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:53.975632 11530 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:58:53.973156 11050 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.016700 10533 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:54.022003 11418 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:54.022913 11418 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:54.023413 11418 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.031443 11050 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.031560 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.032553 11050 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:54.032747 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.033000 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.039047 10533 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:54.033896 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.037580 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.032748 9937 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:54.034013 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.035312 11519 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:54.043135 10010 raft_consensus.cc:2410] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 for term 6 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260430 07:58:54.033660 10009 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:54.046676 10007 raft_consensus.cc:2410] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260430 07:58:54.048082 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.033543 10011 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:54.048962 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.049928 10010 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:54.050146 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.051474 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.051520 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.052312 10011 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:54.052726 10559 raft_consensus.cc:1240] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:54.052920 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.053666 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.053720 10006 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.057713 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.058599 10007 raft_consensus.cc:2393] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:54.033257 11050 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:54.060664 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 102219us
I20260430 07:58:54.064363 11385 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260430 07:58:54.070506 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.072168 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:54.075078 10010 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:58:54.075287 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.080104 9857 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:54.086458 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:54.091248 9903 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:58:54.090067 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 107956us
I20260430 07:58:54.068028 11239 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.067554 11394 raft_consensus.cc:2804] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:54.091085 9904 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:54.093305 10010 raft_consensus.cc:606] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:58:54.095150 11545 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:54.094749 10533 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:54.088470 11543 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.099716 10010 raft_consensus.cc:614] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:58:54.100739 11545 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:54.106268 11545 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.106272 10567 raft_consensus.cc:1240] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:54.108997 9934 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260430 07:58:54.099264 11404 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.110206 11414 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:54.110705 11414 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:54.111096 11414 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:54.111589 11404 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.112080 10006 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:54.114609 11519 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:54.115417 11519 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:54.121542 9828 leader_election.cc:336] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.523s (SENT)
I20260430 07:58:54.122071 9828 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:54.122437 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:54.123463 9828 leader_election.cc:336] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.567s (SENT)
I20260430 07:58:54.123534 11519 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.124521 11512 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.126951 9903 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:54.128075 10003 raft_consensus.cc:2410] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260430 07:58:54.129557 11400 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:54.129624 9904 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:54.130194 11400 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:54.130681 11400 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:54.123932 9828 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:54.131318 11494 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:54.131420 9904 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:54.131752 11512 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.132529 11499 raft_consensus.cc:2749] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:54.132609 11383 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:54.127900 11511 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.136509 9935 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:58:54.129186 9903 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:58:54.148594 9931 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260430 07:58:54.100872 11394 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:54.149648 11394 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.145712 10006 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260430 07:58:54.155355 9978 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:54.156385 11561 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:54.157205 11561 raft_consensus.cc:697] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:54.158344 11561 consensus_queue.cc:237] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:54.174467 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 45322.6us
I20260430 07:58:54.175618 9903 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:54.178411 9903 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:54.156693 11564 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:54.183676 9977 connection.cc:254] RPC call timeout handler was delayed by -1.8782545160000268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
I20260430 07:58:54.133674 11494 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:54.196055 9977 connection.cc:254] RPC call timeout handler was delayed by -1.334696517999987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20260430 07:58:54.192833 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.199985 10006 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:58:54.169651 11418 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:54.191757 9828 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.676s (SENT) [suppressed 36 similar messages]
I20260430 07:58:54.200311 9977 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:54.183825 11542 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:54.191581 11385 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.227767 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.228873 9856 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.208340 11310 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:54.229799 11310 raft_consensus.cc:697] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:54.230403 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.230983 9861 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.230846 11310 consensus_queue.cc:237] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:54.217983 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.234887 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:54.236006 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.239710 11414 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:54.240789 11501 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:54.206773 11418 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.241441 11470 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:54.242004 11470 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.242713 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.243453 9858 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:54.244405 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.245060 11496 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:54.245522 11543 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.198431 11469 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:54.245684 11496 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.246692 11469 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:54.246891 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.246959 11494 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.247747 9854 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.248634 11496 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:54.248800 11469 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:54.222031 9977 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:54.223412 11564 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:54.229658 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.245200 9858 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:54.253659 9863 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.254047 9931 raft_consensus.cc:1240] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:54.254359 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.222230 10559 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260430 07:58:54.245864 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.221630 11323 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:54.242189 9859 raft_consensus.cc:1275] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:58:54.200714 10006 raft_consensus.cc:606] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:58:54.244432 11470 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:54.266284 11400 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.217296 11542 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:54.216907 11050 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:54.272552 9856 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:58:54.262077 11511 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.273860 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.299631 9862 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:54.273976 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.300554 9859 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.275764 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.267635 9855 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:54.281601 9860 raft_consensus.cc:1275] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:58:54.305934 10010 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:58:54.281914 9828 leader_election.cc:336] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.704s (SENT)
I20260430 07:58:54.306514 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.306855 9828 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:54.308228 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:54.308887 9828 leader_election.cc:336] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.775s (SENT)
I20260430 07:58:54.309787 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.306525 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.311259 9855 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:54.312721 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.312872 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.313385 9860 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:54.314091 9855 raft_consensus.cc:2393] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 4: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:54.309671 11512 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.315362 9828 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:54.316656 9855 raft_consensus.cc:1275] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:58:54.317593 11512 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:58:54.314349 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:54.316833 9828 leader_election.cc:336] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.805s (SENT)
I20260430 07:58:54.320858 9828 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:54.322620 11512 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:58:54.329200 9828 leader_election.cc:336] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.834s (SENT)
I20260430 07:58:54.319599 9860 raft_consensus.cc:2393] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 4: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:54.333488 10010 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
W20260430 07:58:54.335701 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 113735us
I20260430 07:58:54.334223 9828 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:54.319545 9855 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:54.282573 11520 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:54.274345 11414 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.348760 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.287766 11542 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:58:54.344893 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 180698us
I20260430 07:58:54.346359 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:58:54.351078 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.351274 9937 raft_consensus.cc:1240] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
W20260430 07:58:54.354825 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.348804 9859 raft_consensus.cc:2410] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:58:54.363634 10010 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:58:54.364343 10010 raft_consensus.cc:606] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:58:54.364998 10010 raft_consensus.cc:614] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:58:54.339062 11323 raft_consensus.cc:697] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:54.345713 11494 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.371186 9858 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:54.375825 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.351563 11512 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:54.336371 11501 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.382164 10533 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:54.367302 11403 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.395382 11323 consensus_queue.cc:237] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:54.399916 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.401804 11512 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.413764 9934 raft_consensus.cc:1240] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
W20260430 07:58:54.414691 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:54.415248 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.417618 9935 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:54.386616 9859 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:54.429837 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.377346 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:54.387763 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:54.440197 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.440649 9858 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:54.448884 9861 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:58:54.440307 9859 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260430 07:58:54.443863 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 114792us
I20260430 07:58:54.386633 11520 raft_consensus.cc:697] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:54.459523 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 45976.9us
I20260430 07:58:54.424937 11564 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:58:54.424784 11050 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:54.433027 11511 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:54.472122 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:54.484385 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.482818 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:54.488705 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.496263 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.496944 9936 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:54.500429 11385 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.501528 9934 raft_consensus.cc:1240] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:54.502388 11501 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:54.492697 11050 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.504856 11050 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.505894 9930 raft_consensus.cc:1240] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:54.509862 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.490146 11470 consensus_queue.cc:1059] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260430 07:58:54.510811 9938 raft_consensus.cc:2393] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:54.480396 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.511538 11476 raft_consensus.cc:3055] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:54.511929 11476 raft_consensus.cc:740] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:54.512002 9855 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.510440 9935 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:54.519198 9933 raft_consensus.cc:1240] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:54.519929 11512 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.512796 11476 consensus_queue.cc:260] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:54.519793 11394 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:54.457715 9856 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:54.521548 10566 raft_consensus.cc:1217] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-3.4] Dedup: 1.3->[3.4-3.4]
I20260430 07:58:54.522135 10566 raft_consensus.cc:1240] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 5. Ops: [1.3-3.4]
I20260430 07:58:54.522150 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.521252 11545 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:54.534734 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 60621.3us
I20260430 07:58:54.535957 9862 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:54.448818 11418 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260430 07:58:54.538349 9863 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:54.538956 9861 raft_consensus.cc:1240] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [1.3-3.4]
W20260430 07:58:54.537693 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 86316.7us
W20260430 07:58:54.541812 9828 leader_election.cc:336] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.859s (SENT)
I20260430 07:58:54.478744 9929 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496"
dest_uuid: "77e19ab653f540339c1c613fad1451b5"
mode: GRACEFUL
new_leader_uuid: "48e81934d52a4e93be5171ceea193032"
from {username='slave'} at 127.0.0.1:52704
I20260430 07:58:54.550103 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:54.551090 9937 raft_consensus.cc:2393] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:54.550017 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.552630 10006 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.545475 9862 raft_consensus.cc:1240] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:54.564545 9828 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:54.564918 10533 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:54.524498 11400 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.536746 9856 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:54.569455 9854 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:54.570348 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.569674 9856 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:58:54.575143 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.576826 11547 raft_consensus.cc:2804] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:54.569855 11476 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.509903 11520 consensus_queue.cc:237] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:54.581066 11336 consensus_queue.cc:1059] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:54.595816 11480 raft_consensus.cc:3055] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Stepping down as leader of term 1
I20260430 07:58:54.573084 11511 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.596299 11480 raft_consensus.cc:740] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:54.598004 11511 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.583474 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.577462 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.586747 9862 raft_consensus.cc:1240] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 1. Current term is 2. Ops: []
I20260430 07:58:54.588989 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.593770 9854 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:54.594158 9857 raft_consensus.cc:1240] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:54.614130 9937 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:58:54.597486 11480 consensus_queue.cc:260] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:54.601713 9931 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.597553 11403 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.573207 9929 raft_consensus.cc:606] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Received request to transfer leadership to TS 48e81934d52a4e93be5171ceea193032
I20260430 07:58:54.588109 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.604355 11545 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.604755 11394 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:54.605551 11542 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:54.606405 11547 raft_consensus.cc:697] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
W20260430 07:58:54.612459 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.633688 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.634562 9862 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:54.626873 9929 raft_consensus.cc:614] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:58:54.642092 9854 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:54.646534 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.647842 9934 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:54.625568 9932 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
W20260430 07:58:54.650581 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 124017us
I20260430 07:58:54.650792 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:54.651350 9828 leader_election.cc:336] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.877s (SENT)
I20260430 07:58:54.651793 9828 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:54.651996 9932 raft_consensus.cc:2393] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:54.652571 11403 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.654881 11336 consensus_queue.cc:1048] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260430 07:58:54.661087 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.653264 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.663442 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.664470 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.665094 10010 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 3
I20260430 07:58:54.665710 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.659621 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.669203 10003 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.582062 9977 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:54.631259 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.655395 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.654695 9978 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:54.670382 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:54.672119 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.616457 9859 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:58:54.653856 9828 leader_election.cc:336] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.888s (SENT)
I20260430 07:58:54.674724 9828 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:54.677501 9937 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:54.624338 10006 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:58:54.679378 9903 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:54.653708 9930 raft_consensus.cc:2393] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:54.692494 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:54.692634 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 110506us
I20260430 07:58:54.693732 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.694316 9862 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:54.682062 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.692834 9978 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:54.696487 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:54.697849 9934 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:54.659559 11587 log.cc:927] Time spent T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.158s user 0.004s sys 0.000s
I20260430 07:58:54.697746 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.699256 9978 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:54.701134 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.705763 11404 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:54.707842 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.708889 9936 raft_consensus.cc:2410] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 5 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260430 07:58:54.677410 11480 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:54.709944 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:54.709971 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.712512 10006 raft_consensus.cc:2393] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 4: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:54.714399 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.715183 9863 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.666462 9932 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:54.666379 11547 consensus_queue.cc:237] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:54.717154 9937 raft_consensus.cc:2393] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:58:54.711617 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 135463us
I20260430 07:58:54.722857 9978 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:54.701567 11561 raft_consensus.cc:2749] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:58:54.724296 9938 raft_consensus.cc:1275] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:54.691804 9856 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:54.704115 11336 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:54.730650 11336 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:54.731107 11336 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.704249 11563 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:54.734383 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.744555 10010 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:54.745294 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:54.745854 9935 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.741003 9936 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:58:54.746855 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 92984.4us
I20260430 07:58:54.744623 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.736259 9978 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:54.738293 11470 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:54.733533 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.711802 11501 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:54.763563 11501 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:54.764076 11501 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:58:54.765547 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.767115 11542 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:54.757535 11470 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:54.738977 9933 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260430 07:58:54.770198 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:54.771229 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:54.739164 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.740206 11563 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:54.772476 10012 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.772756 11470 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.773577 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260430 07:58:54.773993 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.774714 10009 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.775492 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:58:54.776618 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:58:54.777696 9933 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:54.780128 9931 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:54.712685 11403 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.782150 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.782927 9859 raft_consensus.cc:2393] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 3: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:54.782846 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.753364 9937 raft_consensus.cc:2393] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 2: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:54.783835 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.784230 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:54.784365 9862 raft_consensus.cc:1275] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:54.784626 9859 raft_consensus.cc:2410] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:58:54.784754 9937 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.785588 9862 raft_consensus.cc:1275] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:54.785666 10007 raft_consensus.cc:1275] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260430 07:58:54.786067 10007 pending_rounds.cc:85] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 1
I20260430 07:58:54.785991 9855 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:54.786484 10007 pending_rounds.cc:107] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260430 07:58:54.772583 11563 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.786904 10007 raft_consensus.cc:2889] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:54.760810 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:54.788160 9929 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:54.788374 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:54.761382 11404 raft_consensus.cc:697] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:54.761579 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:54.790176 10566 raft_consensus.cc:2393] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:54.790047 11404 consensus_queue.cc:237] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.762246 9759 catalog_manager.cc:5671] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "27f97a330f374849be58a168318c1ec6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:58:54.773316 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:54.794644 9859 raft_consensus.cc:1275] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260430 07:58:54.747952 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:54.756922 11476 raft_consensus.cc:993] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: : Instructing follower 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:58:54.801960 11476 raft_consensus.cc:1081] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Signalling peer 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:58:54.807454 9938 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:54.849419 9759 catalog_manager.cc:5671] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 1 to 3, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 3 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:54.852614 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:54.854341 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 118093us
W20260430 07:58:54.854614 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:54.862267 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 53356.2us
W20260430 07:58:54.863487 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 146250us
I20260430 07:58:54.865011 9978 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:54.853564 9936 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
W20260430 07:58:54.862326 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 98109.4us
W20260430 07:58:54.877743 9828 leader_election.cc:336] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.942s (SENT)
I20260430 07:58:54.878226 9828 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:58:54.882228 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.790022 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:54.899448 11476 consensus_queue.cc:1059] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
W20260430 07:58:54.900264 11606 log.cc:927] Time spent T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.132s user 0.002s sys 0.000s
I20260430 07:58:54.901625 11403 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:54.902092 11520 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:54.903012 9938 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260430 07:58:54.906114 9931 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
W20260430 07:58:54.907948 11596 log.cc:927] Time spent T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.155s user 0.003s sys 0.000s
I20260430 07:58:54.909390 11443 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:58:54.909761 9930 raft_consensus.cc:1240] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
W20260430 07:58:54.916719 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.897851 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.948583 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:54.949368 9860 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.913540 9903 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:54.911936 11561 consensus_queue.cc:1059] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.009s
W20260430 07:58:54.926486 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:54.921401 9932 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:54.942684 11476 raft_consensus.cc:3055] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:54.942961 11520 raft_consensus.cc:697] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:54.977790 11476 raft_consensus.cc:740] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:54.983631 11476 consensus_queue.cc:260] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.984189 11520 consensus_queue.cc:237] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:54.988413 9935 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:54.991868 9863 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:58:54.964701 9978 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:54.961723 11602 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:54.953374 11443 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:54.953624 11394 consensus_queue.cc:1059] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260430 07:58:54.971338 11601 consensus_queue.cc:1059] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260430 07:58:54.993125 11323 raft_consensus.cc:3055] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:58:54.997977 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:58:54.999032 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:54.999423 10010 raft_consensus.cc:1275] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260430 07:58:54.881794 9759 catalog_manager.cc:5671] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
I20260430 07:58:55.035913 10010 pending_rounds.cc:85] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 1
I20260430 07:58:55.036619 10010 pending_rounds.cc:107] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260430 07:58:55.036978 10010 raft_consensus.cc:2889] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:54.970207 11543 consensus_queue.cc:1048] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260430 07:58:55.007406 10012 raft_consensus.cc:2410] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
W20260430 07:58:55.017599 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 197227us
I20260430 07:58:55.018878 10534 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:55.041452 10534 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:54.995848 9761 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48098 (request call id 191) took 10328 ms. Trace:
W20260430 07:58:55.022228 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 57626.6us
I20260430 07:58:55.045943 9978 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:55.022655 9860 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:58:55.049952 11469 raft_consensus.cc:2764] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260430 07:58:55.025456 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 174047us
I20260430 07:58:54.950121 9861 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:55.030875 9933 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:58:55.073782 9935 raft_consensus.cc:1240] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:55.074206 10533 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.073989 9937 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:58:55.075803 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:55.093866 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:55.098495 9828 leader_election.cc:336] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.984s (SENT)
I20260430 07:58:55.099023 9828 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:55.101677 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:55.027616 11403 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:58:55.110224 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 88830us
I20260430 07:58:55.033245 11476 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:55.040740 11565 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:58:55.021903 11443 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:58:55.125945 11520 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:55.104195 11418 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:55.035770 11323 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:55.136425 11239 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:55.137838 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:55.042630 9761 rpcz_store.cc:276] 0430 07:58:44.667436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:44.668867 (+ 1431us) service_pool.cc:225] Handling call
0430 07:58:54.995589 (+10326722us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:58:53.390709 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:58:53.391497 (+ 788us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:58:53.391545 (+ 48us) write_op.cc:435] Acquired schema lock
0430 07:58:53.391584 (+ 39us) tablet.cc:662] Decoding operations
0430 07:58:53.393765 (+ 2181us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:58:53.393884 (+ 119us) write_op.cc:641] Partition lock acquired for write op
0430 07:58:53.394245 (+ 361us) tablet.cc:685] Acquiring locks for 21 operations
0430 07:58:53.404607 (+ 10362us) tablet.cc:701] Row locks acquired
0430 07:58:53.404648 (+ 41us) write_op.cc:260] PREPARE: finished
0430 07:58:53.404831 (+ 183us) write_op.cc:270] Start()
0430 07:58:53.405052 (+ 221us) write_op.cc:276] Timestamp: P: 1777535933404767 usec, L: 0
0430 07:58:53.405102 (+ 50us) op_driver.cc:348] REPLICATION: starting
0430 07:58:53.406294 (+ 1192us) log.cc:844] Serialized 8345 byte log entry
0430 07:58:53.660156 (+253862us) op_driver.cc:464] REPLICATION: finished
0430 07:58:54.001939 (+341783us) write_op.cc:301] APPLY: starting
0430 07:58:54.012710 (+ 10771us) tablet.cc:1367] starting BulkCheckPresence
0430 07:58:54.091712 (+ 79002us) tablet.cc:1370] finished BulkCheckPresence
0430 07:58:54.091865 (+ 153us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:58:54.119847 (+ 27982us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:58:54.120376 (+ 529us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=21
0430 07:58:54.120483 (+ 107us) write_op.cc:312] APPLY: finished
0430 07:58:54.121360 (+ 877us) log.cc:844] Serialized 144 byte log entry
0430 07:58:54.130748 (+ 9388us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:54.130961 (+ 213us) write_op.cc:454] Released schema lock
0430 07:58:54.131045 (+ 84us) write_op.cc:341] FINISH: Updating metrics
0430 07:58:54.197360 (+ 66315us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:54.200221 (+ 2861us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":110,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":287547,"apply.run_cpu_time_us":7129,"apply.run_wall_time_us":186850,"num_ops":21,"prepare.queue_time_us":16207,"prepare.run_cpu_time_us":8492,"prepare.run_wall_time_us":44726,"raft.queue_time_us":159328,"raft.run_cpu_time_us":22604,"raft.run_wall_time_us":230072,"replication_time_us":205290,"thread_start_us":186010,"threads_started":4,"wal-append.queue_time_us":23480,"wal-append.run_cpu_time_us":15673,"wal-append.run_wall_time_us":1695509}]]}
I20260430 07:58:55.139351 11530 consensus_queue.cc:1048] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:55.140625 9904 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.141031 11310 consensus_queue.cc:1048] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:55.142609 9903 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:58:55.143823 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:55.039673 11602 raft_consensus.cc:697] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:55.165676 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1465) took 1130 ms. Trace:
W20260430 07:58:55.198504 11608 log.cc:927] Time spent T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260430 07:58:55.197054 10011 rpcz_store.cc:276] 0430 07:58:54.034835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:54.052944 (+ 18109us) service_pool.cc:225] Handling call
0430 07:58:54.053266 (+ 322us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:54.138045 (+ 84779us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:54.366310 (+228265us) log.cc:844] Serialized 14 byte log entry
0430 07:58:54.600206 (+233896us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:54.630046 (+ 29840us) log.cc:844] Serialized 26 byte log entry
0430 07:58:54.630298 (+ 252us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:54.630349 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:54.630442 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:54.906530 (+276088us) spinlock_profiling.cc:243] Waited 41 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:58:55.149562 (+243032us) raft_consensus.cc:1685] finished
0430 07:58:55.165359 (+ 15797us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:55.165584 (+ 225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10581,"prepare.queue_time_us":217378,"prepare.run_cpu_time_us":1348,"prepare.run_wall_time_us":14439,"raft.queue_time_us":38196,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":11349,"spinlock_wait_cycles":94190336,"thread_start_us":333250,"threads_started":2,"wal-append.queue_time_us":322867}
W20260430 07:58:55.166985 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 118387us
I20260430 07:58:55.170123 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:55.200219 9937 raft_consensus.cc:2393] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:55.177093 9977 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:55.179102 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:55.107023 11394 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:55.122897 11512 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:58:55.202304 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 61739.9us
I20260430 07:58:55.205689 9904 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:55.206053 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 63496.9us
I20260430 07:58:55.094008 11563 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.158890 11565 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:55.204329 9978 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:55.209929 10012 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260430 07:58:55.202703 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 104202us
I20260430 07:58:55.212800 9936 raft_consensus.cc:1275] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:58:55.221091 11050 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:58:55.222319 11511 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:55.223109 11511 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:55.223826 11511 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:58:55.225363 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:55.221871 9759 catalog_manager.cc:5671] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 reported cstate change: term changed from 1 to 3, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 3 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:55.209983 11601 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:55.163925 11403 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:55.241117 11403 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:55.239902 11443 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.243860 10007 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:58:55.245851 11511 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.247476 11443 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:55.248308 11443 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:55.248718 11443 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:55.249066 11443 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:55.251636 11511 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:55.257263 11613 log.cc:927] Time spent T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.154s user 0.002s sys 0.000s
I20260430 07:58:55.163938 11520 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.263408 11404 raft_consensus.cc:3055] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:55.264103 11404 raft_consensus.cc:740] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:55.265810 10928 raft_consensus.cc:3055] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:58:55.266386 10928 raft_consensus.cc:740] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:55.267460 10928 consensus_queue.cc:260] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:55.272984 11602 consensus_queue.cc:237] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.278715 11394 raft_consensus.cc:697] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:55.261020 11520 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:55.282917 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:55.283188 9931 raft_consensus.cc:1275] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260430 07:58:55.284623 11480 consensus_meta.cc:287] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.566s user 0.011s sys 0.000s
I20260430 07:58:55.201741 9856 raft_consensus.cc:3060] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:58:55.298250 10009 consensus_meta.cc:287] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.523s user 0.014s sys 0.003s
I20260430 07:58:55.298691 10009 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:58:55.298933 9861 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:58:55.283002 11609 log.cc:927] Time spent T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20260430 07:58:55.280207 11394 consensus_queue.cc:237] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:55.303298 10003 consensus_meta.cc:287] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.606s user 0.014s sys 0.005s
I20260430 07:58:55.306334 10003 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:58:55.131932 11586 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:55.218809 11565 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:58:55.318523 9828 leader_election.cc:336] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.041s (SENT)
I20260430 07:58:55.319226 9828 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:58:55.305294 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 99609.8us
I20260430 07:58:55.320749 9904 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.252636 11501 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.299612 10012 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:58:55.280253 11520 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:55.307696 9929 consensus_meta.cc:287] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.519s user 0.019s sys 0.001s
W20260430 07:58:55.323516 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 214307us
I20260430 07:58:55.329321 9904 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:55.331961 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:55.311496 11493 raft_consensus.cc:2764] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260430 07:58:55.339326 9935 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260430 07:58:55.270901 11404 consensus_queue.cc:260] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:55.337260 11336 consensus_meta.cc:287] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.606s user 0.012s sys 0.007s
I20260430 07:58:55.333076 11543 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:58:55.326026 11586 raft_consensus.cc:697] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
W20260430 07:58:55.346541 11470 consensus_meta.cc:287] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.572s user 0.010s sys 0.010s
W20260430 07:58:55.344564 9854 consensus_meta.cc:287] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.702s user 0.011s sys 0.007s
I20260430 07:58:55.332569 9929 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
W20260430 07:58:55.373689 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 227837us
I20260430 07:58:55.354893 11512 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:55.380023 9977 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:55.357792 11542 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260430 07:58:55.391077 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 62080.9us
I20260430 07:58:55.333371 11520 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.394991 9904 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.366448 11239 raft_consensus.cc:697] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:55.378710 11404 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:55.422854 11586 consensus_queue.cc:237] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:55.377512 11470 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.427434 11470 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:55.429147 9856 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:58:55.322539 11323 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.400131 11561 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader election won for term 2
I20260430 07:58:55.431793 11561 raft_consensus.cc:697] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:55.432785 11561 consensus_queue.cc:237] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.433454 11323 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:58:55.403290 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 100218us
I20260430 07:58:55.406000 11469 consensus_queue.cc:1059] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:55.409971 11443 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:55.399302 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 194928us
I20260430 07:58:55.414433 11239 consensus_queue.cc:237] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:55.417984 11612 log.cc:927] Time spent T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20260430 07:58:55.418553 9854 raft_consensus.cc:1275] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:58:55.421090 11563 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:55.452963 9934 consensus_meta.cc:287] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.751s user 0.009s sys 0.015s
W20260430 07:58:55.454139 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 52963.8us
I20260430 07:58:55.469501 9903 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:55.379815 11615 consensus_queue.cc:1059] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.088s
I20260430 07:58:55.428684 11470 consensus_queue.cc:1059] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260430 07:58:55.369248 11336 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.402892 11542 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260430 07:58:55.471860 10008 raft_consensus.cc:1275] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:58:55.473301 9934 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:58:55.459483 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 64607.6us
I20260430 07:58:55.474792 9861 raft_consensus.cc:1275] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:55.475982 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:55.475901 9747 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 48e81934d52a4e93be5171ceea193032 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:55.476621 11601 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:55.480592 9863 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68"
dest_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:37838
I20260430 07:58:55.492651 9863 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:58:55.493088 9863 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:55.467032 11511 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:58:55.482527 11310 consensus_queue.cc:1059] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260430 07:58:55.482905 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:55.486071 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:55.486321 11530 consensus_queue.cc:1059] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:58:55.487157 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:55.481721 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:55.501186 9859 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:58:55.493822 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 96298.8us
I20260430 07:58:55.472563 10928 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:55.498952 9860 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:55.494359 9904 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:55.523972 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 107910us
I20260430 07:58:55.530395 11310 consensus_queue.cc:1243] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer 48e81934d52a4e93be5171ceea193032 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:58:55.486125 11520 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:55.490356 11418 raft_consensus.cc:697] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:55.553757 9862 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:55.547236 9863 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:55.561412 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 67123.8us
I20260430 07:58:55.562451 9904 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.563431 11599 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:58:55.565909 9863 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:55.504784 11619 raft_consensus.cc:3055] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
W20260430 07:58:55.573375 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:55.540732 9857 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:58:55.560423 9932 consensus_meta.cc:287] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.559s user 0.011s sys 0.000s
I20260430 07:58:55.586294 9859 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:58:55.543001 11601 raft_consensus.cc:697] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:55.591284 10566 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260430 07:58:55.611276 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:55.611315 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 48960us
I20260430 07:58:55.612250 9904 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:58:55.613413 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 88166.9us
I20260430 07:58:55.614086 9977 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:58:55.618404 9860 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:55.558589 11501 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:55.630537 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 84102.7us
I20260430 07:58:55.633077 9904 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.635015 9904 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.552680 11050 consensus_queue.cc:1048] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.069s
I20260430 07:58:55.578896 11619 raft_consensus.cc:740] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:55.640547 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 93728.1us
I20260430 07:58:55.651660 10008 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:58:55.556172 11621 consensus_queue.cc:1059] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
I20260430 07:58:55.624877 11543 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:58:55.610989 9932 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:58:55.612854 11310 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
W20260430 07:58:55.689616 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:55.650878 11385 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:55.691299 11512 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:55.696390 11443 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:55.654419 11542 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260430 07:58:55.757012 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 100485us
I20260430 07:58:55.635994 11520 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:55.654621 11599 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:58:55.797118 9904 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.814643 9857 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:58:55.817310 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:55.810791 11385 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:58:55.863340 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 200493us
W20260430 07:58:55.871909 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 79232.2us
I20260430 07:58:55.877743 9904 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.839943 11542 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:55.863086 11239 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:55.867996 11599 raft_consensus.cc:697] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:55.866043 11470 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:55.865388 11385 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:55.721064 11601 consensus_queue.cc:237] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:55.917153 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 461507us
I20260430 07:58:55.701592 11418 consensus_queue.cc:237] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:55.867367 11336 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:55.862262 11404 consensus_queue.cc:1059] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.050s
I20260430 07:58:55.869694 11632 consensus_queue.cc:1059] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20260430 07:58:55.913645 11542 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:55.839237 11512 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.884155 11530 raft_consensus.cc:2804] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:55.857080 11520 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:55.931638 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 269798us
W20260430 07:58:55.940675 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 127311us
I20260430 07:58:55.940785 10533 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.938692 11310 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260430 07:58:55.947379 9828 leader_election.cc:336] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.117s (SENT)
I20260430 07:58:55.972828 9828 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:55.918474 11476 consensus_queue.cc:1059] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20260430 07:58:55.946978 11470 raft_consensus.cc:697] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:55.957422 11639 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:55.976559 11639 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.976789 11470 consensus_queue.cc:237] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.977702 11602 raft_consensus.cc:3055] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:58:55.978152 11602 raft_consensus.cc:740] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:55.978964 11602 consensus_queue.cc:260] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.979528 11639 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:55.980065 11476 raft_consensus.cc:3055] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:55.980245 11602 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:58:55.980523 11476 raft_consensus.cc:740] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:55.981884 11630 log.cc:927] Time spent T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.154s user 0.000s sys 0.001s
I20260430 07:58:55.981514 11476 consensus_queue.cc:260] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:55.975464 11643 raft_consensus.cc:2804] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:55.983945 11476 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:55.984066 11643 raft_consensus.cc:697] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
W20260430 07:58:55.968120 11565 consensus_meta.cc:287] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.633s user 0.010s sys 0.003s
I20260430 07:58:55.927963 11239 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:56.015095 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 137344us
I20260430 07:58:56.016458 9904 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:55.995101 11530 raft_consensus.cc:697] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:58:56.031632 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:55.985563 11643 consensus_queue.cc:237] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.4, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:56.042501 10567 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260430 07:58:55.997730 11632 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260430 07:58:56.043134 9904 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:56.046591 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:56.055308 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:56.056211 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:56.056950 9863 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:56.050184 10559 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:56.066058 9977 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:56.050819 11637 raft_consensus.cc:3055] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:56.013545 11639 consensus_queue.cc:1048] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260430 07:58:56.007603 11565 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.075625 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:56.076244 9860 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:58:56.074899 9857 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:58:56.098716 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 55672.1us
I20260430 07:58:56.099647 9904 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:56.108286 10004 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:58:56.084558 11310 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:58:56.070348 11648 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:56.071519 11637 raft_consensus.cc:740] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:56.114535 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 167105us
I20260430 07:58:56.115828 11648 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:56.118706 11239 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:56.104267 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:58:56.171146 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:56.235183 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:56.066675 11640 consensus_queue.cc:1059] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260430 07:58:56.125744 11530 consensus_queue.cc:237] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.084359 11650 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:56.102222 9929 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260430 07:58:56.110954 11658 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:56.101495 11621 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:56.005950 11599 consensus_queue.cc:237] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:56.242089 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 176037us
W20260430 07:58:56.247555 9828 leader_election.cc:336] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.151s (SENT)
I20260430 07:58:56.241392 11547 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:58:56.253934 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:56.264212 9828 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:56.256114 9929 pending_rounds.cc:85] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 5
I20260430 07:58:56.247924 11310 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:58:56.254602 10928 consensus_meta.cc:287] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.711s user 0.015s sys 0.000s
I20260430 07:58:56.248090 11050 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:56.270313 9977 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:58:56.247591 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 148003us
W20260430 07:58:56.259994 11323 consensus_meta.cc:287] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.826s user 0.011s sys 0.008s
I20260430 07:58:56.256079 9856 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:56.263904 11512 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:56.284564 11648 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:56.264317 11653 log.cc:927] Time spent T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.203s user 0.004s sys 0.003s
I20260430 07:58:56.250310 11652 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:56.272255 9929 pending_rounds.cc:107] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 3.6
W20260430 07:58:56.292236 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 229741us
I20260430 07:58:56.272039 11657 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:56.277326 9904 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:56.270905 9863 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:56.301323 10005 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260430 07:58:56.273377 11520 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:56.287576 11658 raft_consensus.cc:2749] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:56.302121 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:56.307667 9904 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:56.309213 10008 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:58:56.291745 11470 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:56.291649 11476 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:58:56.293224 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 266617us
I20260430 07:58:56.290855 11512 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:56.294857 9929 raft_consensus.cc:2889] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:56.274427 11547 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:56.292135 11637 consensus_queue.cc:260] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.338056 11547 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:56.338533 11637 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:56.305512 11657 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.260113 11650 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:56.307199 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 245902us
I20260430 07:58:56.295143 11050 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.298941 11621 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.032s
I20260430 07:58:56.321746 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:56.326452 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:56.340077 11512 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.334936 11542 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.310461 11310 raft_consensus.cc:2749] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:56.314554 11652 consensus_queue.cc:1048] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260430 07:58:56.334293 9931 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260430 07:58:56.316042 11565 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:56.395902 9978 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:56.396680 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 86706.3us
I20260430 07:58:56.305869 11648 consensus_queue.cc:1059] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260430 07:58:56.324056 11618 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:56.407750 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:56.411083 11239 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260430 07:58:56.422611 10533 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:56.415093 9929 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260430 07:58:56.416915 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:58:56.425949 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 96597.5us
I20260430 07:58:56.438190 9978 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:56.348407 11659 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:58:56.357651 10559 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:56.407547 9932 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:58:56.363088 11602 consensus_queue.cc:1048] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260430 07:58:56.358982 11385 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.339828 11470 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:56.436885 11648 consensus_queue.cc:1243] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer d66db415f0f0402fa431349afbf59888 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260430 07:58:56.450148 9978 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:56.405371 11520 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260430 07:58:56.455418 9978 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:56.456597 11655 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:58:56.424387 11662 consensus_queue.cc:1048] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260430 07:58:56.466522 11632 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:56.467015 11632 raft_consensus.cc:697] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:56.467262 9978 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:56.445866 11565 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:56.445539 11394 raft_consensus.cc:2804] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:56.459540 11542 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:56.459725 11385 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:56.448640 11530 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:56.460683 11657 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:56.461730 11050 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:56.462368 11512 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:56.465442 11664 log.cc:927] Time spent T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.146s user 0.000s sys 0.001s
I20260430 07:58:56.466043 11655 raft_consensus.cc:697] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:56.457217 11476 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:56.451448 11641 consensus_queue.cc:1048] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.188s
I20260430 07:58:56.487739 11476 raft_consensus.cc:697] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:56.488678 11476 consensus_queue.cc:237] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.489405 11667 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260430 07:58:56.490021 11470 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:56.458230 11650 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:56.491333 11651 log.cc:927] Time spent T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.203s user 0.000s sys 0.001s
I20260430 07:58:56.467923 11632 consensus_queue.cc:237] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.497797 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:56.498957 10566 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:56.437152 11618 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:56.503484 11650 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:56.475594 11565 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:56.479077 11336 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:56.508821 11336 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:56.509167 11618 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:56.509402 11336 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:56.481307 11530 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:56.514274 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 68420.8us
I20260430 07:58:56.510883 11565 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:56.504030 11650 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.533756 11655 consensus_queue.cc:237] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.537948 11530 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:56.548261 11650 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:56.726897 11512 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:56.727674 11512 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:56.771927 11650 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:56.935978 11657 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:56.973362 11512 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:56.951943 11565 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.975922 11657 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:57.050159 10559 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.000s sys 0.010s
W20260430 07:58:57.061591 10559 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s user 0.000s sys 0.010s
I20260430 07:58:57.062695 11511 raft_consensus.cc:3055] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
W20260430 07:58:57.062963 11678 log.cc:927] Time spent T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260430 07:58:57.063264 11511 raft_consensus.cc:740] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:57.064209 11511 consensus_queue.cc:260] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.065784 11511 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:58:57.066215 11336 consensus_meta.cc:287] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.549s user 0.008s sys 0.001s
I20260430 07:58:57.066669 11336 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:56.995380 11650 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.109242 11619 consensus_queue.cc:260] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.111411 11619 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:57.138235 11565 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:57.137702 11520 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.001s sys 0.006s
W20260430 07:58:57.144426 11520 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s user 0.001s sys 0.006s
I20260430 07:58:57.149590 11650 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:57.147991 11657 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:57.155791 11336 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:57.156284 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.156890 11663 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.157439 11663 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.159847 11599 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.160404 11599 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.166098 11511 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:57.166826 11511 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.170094 11511 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:57.176374 11599 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.178014 11663 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:57.178555 9903 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.829s user 0.012s sys 0.037s
W20260430 07:58:57.179026 9903 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s user 0.013s sys 0.037s
I20260430 07:58:57.200477 11663 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.201149 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:57.201051 11663 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.202522 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:57.203083 11663 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:57.204789 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.213222 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:57.215281 9933 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:57.216960 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:57.218784 9933 raft_consensus.cc:2393] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:58:57.219231 11394 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:57.219892 11394 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:57.219931 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:57.230252 9933 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:57.231433 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:57.233500 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:57.234174 9933 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:57.237208 11563 raft_consensus.cc:3055] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:57.237879 11563 raft_consensus.cc:740] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:57.238750 11563 consensus_queue.cc:260] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.240097 11563 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.242815 9930 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260430 07:58:57.244052 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:57.244114 9933 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:57.245381 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:57.244741 9930 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:57.244773 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:57.249516 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:57.250336 9929 raft_consensus.cc:2410] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260430 07:58:57.250152 11511 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.250545 9930 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.251098 11511 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.252777 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:57.253348 11511 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.253496 9929 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.256336 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:57.257261 11635 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:57.258137 11657 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.258667 11657 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.259109 11635 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:58:57.260867 11657 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.256407 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:57.261513 11682 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:58:57.262652 9933 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:57.265182 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:57.265360 11511 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.266110 9933 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:58:57.266151 11511 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.270707 11336 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.271149 11520 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.271486 11336 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.277199 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.272924 11520 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.279206 11336 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:57.280328 11520 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:57.277220 11601 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.281652 11601 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.282624 11469 raft_consensus.cc:3055] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:57.283090 9857 raft_consensus.cc:1275] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:57.283643 11601 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.284444 9857 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
W20260430 07:58:57.285372 11631 log.cc:927] Time spent T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 1.464s user 0.006s sys 0.001s
I20260430 07:58:57.283089 11469 raft_consensus.cc:740] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:57.286490 9933 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:57.286648 11469 consensus_queue.cc:260] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.287729 11469 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.287786 9933 raft_consensus.cc:1240] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
I20260430 07:58:57.288755 9930 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:57.290376 9930 raft_consensus.cc:1275] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:58:57.294672 11511 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.294965 11394 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.298105 11394 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:57.304484 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.305482 9929 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:57.307425 9929 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260430 07:58:57.309384 9929 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260430 07:58:57.309782 9857 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260430 07:58:57.311787 9929 raft_consensus.cc:1240] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:57.293126 9933 raft_consensus.cc:1240] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:58:57.313169 11394 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:58:57.313823 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:57.314568 9857 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:57.312814 9930 raft_consensus.cc:1275] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:57.317369 9929 raft_consensus.cc:1275] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260430 07:58:57.317759 9936 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260430 07:58:57.318192 9936 pending_rounds.cc:85] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 5
I20260430 07:58:57.318459 9936 pending_rounds.cc:107] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260430 07:58:57.318773 9936 raft_consensus.cc:2889] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260430 07:58:57.321520 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 60340.8us
I20260430 07:58:57.322863 9935 raft_consensus.cc:1275] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:57.323726 11394 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.002s
I20260430 07:58:57.330191 9854 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260430 07:58:57.332398 10928 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:58:57.336166 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:57.336270 10533 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:57.336812 10003 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:57.341176 9857 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:58:57.342706 11512 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:57.343322 11512 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.344313 10003 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:58:57.345448 11657 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:57.346037 11657 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.348135 11657 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.342021 11650 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:57.349684 10928 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:58:57.351632 11512 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.357681 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:57.359778 9936 raft_consensus.cc:3060] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:57.360841 10534 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:58:57.370854 11623 raft_consensus.cc:2804] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:57.350555 11650 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.371675 11623 raft_consensus.cc:697] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:58:57.373171 11650 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.372859 11623 consensus_queue.cc:237] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.380676 9861 raft_consensus.cc:1275] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:58:57.385934 9861 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260430 07:58:57.390295 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.392997 9936 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:58:57.397415 11404 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.403601 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:57.398232 11654 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.397186 11618 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260430 07:58:57.414702 11654 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.419595 11654 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:57.412803 11404 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.424259 11404 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.424408 11615 raft_consensus.cc:3055] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:57.426455 11615 raft_consensus.cc:740] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:57.428622 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:57.430016 10559 raft_consensus.cc:2410] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260430 07:58:57.432392 11615 consensus_queue.cc:260] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.459916 11616 raft_consensus.cc:2955] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:57.464381 11615 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.464082 9861 raft_consensus.cc:2955] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
W20260430 07:58:57.504745 9856 consensus_meta.cc:287] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 1.213s user 0.019s sys 0.007s
I20260430 07:58:57.505223 9856 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:57.512055 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1252) took 1468 ms. Trace:
I20260430 07:58:57.512674 9856 rpcz_store.cc:276] 0430 07:58:56.044022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:56.045498 (+ 1476us) service_pool.cc:225] Handling call
0430 07:58:57.511984 (+1466486us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:57.500944 11561 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:57.515333 11561 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.521206 11561 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:57.529557 11647 raft_consensus.cc:3055] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:58:57.530329 11647 raft_consensus.cc:740] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:57.531565 11647 consensus_queue.cc:260] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.538839 11647 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:57.540126 10007 raft_consensus.cc:3055] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:58:57.540571 10007 raft_consensus.cc:740] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:57.544301 10007 consensus_queue.cc:260] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.545553 10007 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:58:57.550215 9904 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.110s user 0.006s sys 0.003s
W20260430 07:58:57.554585 9904 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.115s user 0.007s sys 0.003s
I20260430 07:58:57.557318 11404 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.557895 11404 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.562153 11654 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:57.564572 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.569495 9861 raft_consensus.cc:2410] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:58:57.570677 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.568673 11654 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.576299 10007 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260430 07:58:57.576695 10007 pending_rounds.cc:85] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 4
I20260430 07:58:57.576893 10007 pending_rounds.cc:107] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260430 07:58:57.577039 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 166) took 1071 ms. Trace:
I20260430 07:58:57.577183 10007 raft_consensus.cc:2889] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:57.577963 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 148) took 1272 ms. Trace:
I20260430 07:58:57.578128 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:57.578349 11494 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.578876 9854 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.578388 10007 rpcz_store.cc:276] 0430 07:58:56.305182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:56.308075 (+ 2893us) service_pool.cc:225] Handling call
0430 07:58:56.329740 (+ 21665us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:57.540018 (+1210278us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54000d69a0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5691fb1a0 00007fb5692855a0 00007fb569286611 00007fb56928316f 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c
0430 07:58:57.577719 (+ 37701us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:57.577899 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":295953792}
I20260430 07:58:57.579062 11494 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.580857 11623 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:58:57.582141 11511 consensus_queue.cc:1048] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:57.584414 9934 raft_consensus.cc:2955] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:57.589828 11404 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:57.590418 11561 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.590569 9861 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:57.590986 11561 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.591959 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.592650 9861 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 1.
I20260430 07:58:57.567399 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:57.593559 9856 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.577457 9860 rpcz_store.cc:276] 0430 07:58:56.505952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:57.576162 (+1070210us) service_pool.cc:225] Handling call
0430 07:58:57.576553 (+ 391us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:57.576729 (+ 176us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:57.576754 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:57.576797 (+ 43us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:57.576825 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:57.576875 (+ 50us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:57.576983 (+ 108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:57.597424 11563 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:57.598366 11443 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:57.598914 11443 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.599334 11654 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.599403 11494 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:57.598155 11563 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:57.600579 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.601367 11563 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:57.602419 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.598927 11561 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:57.603006 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:57.603832 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:57.604501 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.605458 11443 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:57.606199 11563 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:57.611131 10005 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:58:57.610795 11563 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:57.612733 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:57.613149 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.615190 11563 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:57.615700 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:57.607115 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 149) took 1200 ms. Trace:
I20260430 07:58:57.607151 9860 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:58:57.617257 10009 rpcz_store.cc:276] 0430 07:58:56.406891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:56.410059 (+ 3168us) service_pool.cc:225] Handling call
0430 07:58:56.410412 (+ 353us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:56.410822 (+ 410us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:56.410848 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:56.411351 (+ 503us) log.cc:844] Serialized 26 byte log entry
0430 07:58:56.468440 (+ 57089us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:58:56.468473 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:56.468550 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:57.606863 (+1138313us) raft_consensus.cc:1685] finished
0430 07:58:57.606888 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:57.607055 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":56706,"threads_started":1,"wal-append.queue_time_us":108303,"wal-append.run_cpu_time_us":1760,"wal-append.run_wall_time_us":1010736}
I20260430 07:58:57.619819 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.620157 9978 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:57.620369 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 48656.8us
I20260430 07:58:57.621006 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:57.621568 9978 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:57.621585 9861 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:58:57.628504 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:57.647663 9854 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:57.647791 9856 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:57.653124 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.607476 11561 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.654695 10005 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:57.649219 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:57.655058 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.653570 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:57.658748 9861 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:58:57.660845 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:57.643499 11443 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:57.607288 11640 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:57.661972 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.643136 11563 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:57.678141 11563 raft_consensus.cc:697] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:57.679461 11563 consensus_queue.cc:237] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.659327 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.681860 10011 raft_consensus.cc:2393] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:57.660980 9860 raft_consensus.cc:2393] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:58:57.682788 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.683460 10011 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:57.684396 11640 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.651104 11654 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20260430 07:58:57.686909 11640 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:57.662972 9854 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.672026 9859 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
W20260430 07:58:57.688234 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.688311 10005 raft_consensus.cc:1275] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:58:57.688654 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1393) took 1182 ms. Trace:
I20260430 07:58:57.663096 9857 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:57.674170 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.676942 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.689745 9978 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:57.662825 11561 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.678066 9861 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:57.646612 11511 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.691943 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.695730 9861 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:57.689214 9859 rpcz_store.cc:276] 0430 07:58:56.506272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:57.671489 (+1165217us) service_pool.cc:225] Handling call
0430 07:58:57.671846 (+ 357us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:57.688315 (+ 16469us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:57.688581 (+ 266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:57.696975 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.697357 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.698036 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.699265 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.691488 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.699527 9859 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:57.700390 9978 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:57.702550 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.703008 9857 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:57.703210 9861 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:57.703286 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.703981 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.704610 9857 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:58:57.720145 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.741604 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:58:57.741842 9860 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:58:57.725165 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.742569 9937 raft_consensus.cc:2410] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260430 07:58:57.734373 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.712282 9978 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:57.712414 11336 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:58:57.704274 10010 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:57.713608 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.695976 11443 raft_consensus.cc:697] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:57.710867 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.699198 11615 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:58:57.705153 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.749223 10011 raft_consensus.cc:2410] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:58:57.702054 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.689224 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:57.754135 9859 raft_consensus.cc:2393] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 4: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:58:57.751348 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.751977 9856 raft_consensus.cc:2410] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:58:57.755971 9854 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:57.751359 9861 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:58:57.756364 9978 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:57.757853 9860 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:58:57.760618 9861 raft_consensus.cc:1275] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:58:57.760924 9860 raft_consensus.cc:1275] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:58:57.762367 9860 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 1 FOLLOWER]: Advancing to term 2
I20260430 07:58:57.762780 9861 raft_consensus.cc:1240] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
W20260430 07:58:57.763350 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.764174 9861 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260430 07:58:57.765220 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.731467 11561 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:57.773252 9856 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260430 07:58:57.774001 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:57.776635 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.778869 9857 raft_consensus.cc:2410] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260430 07:58:57.778869 10011 raft_consensus.cc:2410] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260430 07:58:57.750860 11336 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:57.759127 9859 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:58:57.785001 9857 raft_consensus.cc:1275] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260430 07:58:57.755999 11640 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:57.785704 9857 pending_rounds.cc:85] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 1
I20260430 07:58:57.793972 9857 pending_rounds.cc:107] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260430 07:58:57.702353 11511 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.794628 9857 raft_consensus.cc:2889] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:58:57.796130 9978 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:57.791447 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.800836 9856 raft_consensus.cc:1275] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:58:57.801167 9859 raft_consensus.cc:1275] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:58:57.804921 9860 raft_consensus.cc:1275] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260430 07:58:57.779119 11615 raft_consensus.cc:697] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:57.806802 9978 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:57.808102 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.808967 10011 raft_consensus.cc:2410] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260430 07:58:57.786154 11336 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:58:57.811661 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.812469 9978 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:57.818235 9904 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:57.820202 9904 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:57.826407 9978 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:57.793439 11640 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:57.798069 11654 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.827965 9904 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:57.805225 11626 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:57.814298 11561 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:57.837675 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:57.832496 11494 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:58:57.833766 11640 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.830893 11647 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:57.830621 11520 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:57.834945 11512 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.792375 11443 consensus_queue.cc:237] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.839699 11561 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:57.855123 11494 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:57.836921 11654 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.856755 11647 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:57.865928 11561 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:57.868938 11615 consensus_queue.cc:237] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.858426 11520 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:57.863945 11512 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.893565 9931 raft_consensus.cc:1275] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260430 07:58:57.896399 11708 log.cc:927] Time spent T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260430 07:58:57.899492 11654 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.900177 11654 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:58:57.900534 11654 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:57.900952 11654 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.904516 11511 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.873451 11647 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:57.867704 11494 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:58:57.905686 11414 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:57.880371 11520 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:57.910349 11336 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.942610 11512 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.943387 11336 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:57.943989 11336 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:57.944442 11336 raft_consensus.cc:697] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:57.945231 11520 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.947974 11520 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:57.949535 11520 consensus_queue.cc:1048] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:57.911320 9861 raft_consensus.cc:1275] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260430 07:58:57.953110 10005 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
W20260430 07:58:57.970218 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 142370us
I20260430 07:58:57.970551 11729 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:57.971262 9904 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:58:57.971333 11729 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.974473 11729 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:57.975195 11729 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:58:57.978511 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 146949us
I20260430 07:58:57.986320 11626 raft_consensus.cc:697] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:57.987150 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 139011us
I20260430 07:58:57.994375 11637 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:57.999496 11336 consensus_queue.cc:237] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:57.955349 11561 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:58.030445 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:58.006291 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 46604.8us
W20260430 07:58:58.056152 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:58.104977 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:57.989886 11626 consensus_queue.cc:237] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:58.127099 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.005220 11561 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:58.152307 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 165946us
W20260430 07:58:58.161042 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:58.161044 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:58.193979 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.006481 11718 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.195s
I20260430 07:58:58.113027 11647 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:58.205612 11647 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:58.206485 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 53328.4us
I20260430 07:58:58.006217 11494 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:58.360110 11640 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:58.362660 11640 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:58:58.363348 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.256958 11722 consensus_queue.cc:1059] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.211s
W20260430 07:58:58.523016 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:58.523977 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 534974us
I20260430 07:58:58.524312 11494 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:58.525013 9903 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:58.526081 9904 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:58.527109 11336 raft_consensus.cc:3055] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:58.527509 11336 raft_consensus.cc:740] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:58.529587 11512 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:58.530072 11512 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:58.532276 11512 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:58.528333 11336 consensus_queue.cc:260] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:58.540781 11336 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:58.552973 11494 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:58.583977 9903 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:58.587724 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 61712us
I20260430 07:58:58.588891 9904 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:58.591557 9904 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:58:58.596369 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.602528 9903 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:58:58.572966 11586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.000s sys 0.012s
W20260430 07:58:58.616211 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:58.626507 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 418246us
I20260430 07:58:58.580730 11722 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:58:58.611562 11721 log.cc:927] Time spent T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260430 07:58:58.622746 11586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s user 0.000s sys 0.013s
I20260430 07:58:58.619613 11543 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260430 07:58:58.636485 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.610749 11647 consensus_queue.cc:1048] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260430 07:58:58.639171 11543 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:58.657112 11626 consensus_queue.cc:1048] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:58:58.657696 11543 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:58:58.665364 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.670039 11647 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:58:58.670995 9828 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.397s user 0.044s sys 0.072s
I20260430 07:58:58.634477 11724 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:58.635916 11722 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:58.684834 11732 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:58.685470 11732 raft_consensus.cc:697] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:58.686518 11732 consensus_queue.cc:237] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:58.689333 11730 consensus_queue.cc:1059] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260430 07:58:58.690702 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 163) took 2194 ms. Trace:
I20260430 07:58:58.691082 11722 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:58.691113 9863 rpcz_store.cc:276] 0430 07:58:56.504791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:58.689770 (+2184979us) service_pool.cc:225] Handling call
0430 07:58:58.690133 (+ 363us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:58.690346 (+ 213us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:58.690377 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:58:58.690428 (+ 51us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:58:58.690465 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:58.690522 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:58:58.690650 (+ 128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:58.683504 11724 raft_consensus.cc:697] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:58.671069 11647 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:58:58.682452 11654 consensus_meta.cc:287] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.755s user 0.006s sys 0.007s
W20260430 07:58:58.700130 9828 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.426s user 0.044s sys 0.072s
I20260430 07:58:58.702373 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:58:58.714129 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 111686us
W20260430 07:58:58.733325 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.665215 11626 consensus_queue.cc:1048] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260430 07:58:58.738996 9903 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:58.740372 9863 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:58:58.743213 9903 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:58:58.752853 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:58.753152 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:58.753533 9857 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:58:58.753712 9861 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 4
I20260430 07:58:58.754766 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:58.748901 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:58.756114 9859 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 4
I20260430 07:58:58.761570 9863 raft_consensus.cc:1275] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:58:58.781126 9856 raft_consensus.cc:1275] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:58:58.785486 9857 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:58:58.802095 9978 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:58.798673 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 207114us
I20260430 07:58:58.736176 11722 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260430 07:58:58.803439 9904 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:58:58.808218 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 2.55924e+06us
I20260430 07:58:58.804044 9859 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:58.782176 11336 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:58.744220 11654 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:58.824508 11735 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:58.825223 11735 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:58.825971 11733 log.cc:927] Time spent T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
I20260430 07:58:58.827705 11735 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:58:58.827975 11731 log.cc:927] Time spent T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20260430 07:58:58.811584 9856 raft_consensus.cc:1275] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260430 07:58:58.834084 9828 leader_election.cc:336] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.388s (SENT)
I20260430 07:58:58.810153 9904 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:58.860419 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.860541 9978 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:58.866973 9978 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:58.862049 9856 raft_consensus.cc:1275] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:58:58.840854 9828 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:58:58.863134 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:58.868507 9859 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:58.872321 9861 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:58:58.811570 11640 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:58:58.873814 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:58.890026 11512 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:58.883056 11443 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:58:58.871018 11735 consensus_queue.cc:1048] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.033s
I20260430 07:58:58.820027 11732 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:58:58.868206 11727 log.cc:927] Time spent T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20260430 07:58:58.896927 9856 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:58:58.898020 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 72498.4us
I20260430 07:58:58.906471 9904 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:58:58.837783 11730 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:58:58.917670 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 174441us
I20260430 07:58:58.882042 11640 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:58:58.931841 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.899452 11561 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
W20260430 07:58:58.916620 11476 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.939s user 0.016s sys 0.035s
I20260430 07:58:58.901930 11443 raft_consensus.cc:697] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:58.880934 11626 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.050s
I20260430 07:58:58.940855 9904 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:58:58.935458 11640 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:58:58.941411 11476 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.966s user 0.017s sys 0.036s
W20260430 07:58:58.951892 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 129109us
I20260430 07:58:58.931877 11741 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:58:58.938844 11732 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:58.821753 11724 consensus_queue.cc:237] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:58.837143 11336 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:58.961794 11739 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:58.977921 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:58.947181 11561 raft_consensus.cc:697] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:58:58.968324 10003 raft_consensus.cc:1275] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260430 07:58:58.985916 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:58.986541 10003 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 5
I20260430 07:58:58.972349 10010 raft_consensus.cc:1275] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260430 07:58:58.990090 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:58.990700 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:58.991060 10010 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:58:58.992839 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:58.982524 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.971367 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:58.994987 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:58.983888 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:58.997129 10007 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:59.002511 9903 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:58:59.012625 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.962103 11742 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:58:58.961129 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 94136.5us
I20260430 07:58:59.016080 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:58.972697 11512 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:58.985939 11739 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:59.018576 11512 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.018781 11561 consensus_queue.cc:237] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:58.966202 11748 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260430 07:58:59.017447 9828 connection.cc:254] RPC call timeout handler was delayed by -4.9755468650000125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1878s for a response
I20260430 07:58:59.015027 11742 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:59.013000 11754 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:58:59.016358 9978 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:58:59.075417 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 134574us
I20260430 07:58:59.031541 10007 raft_consensus.cc:1240] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
W20260430 07:58:59.034226 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:58.970254 11336 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:59.077790 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:58.973717 11732 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.035450 10003 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:58:59.076450 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:59.079321 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 85211.7us
W20260430 07:58:59.081604 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 81484.5us
I20260430 07:58:59.038383 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:59.103137 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:59.079442 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:59.031920 9978 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:58.972911 11730 consensus_queue.cc:1048] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20260430 07:58:59.137246 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.145407 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.142349 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.147146 10007 raft_consensus.cc:2393] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:59.021651 11748 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:59.080121 11725 consensus_queue.cc:1048] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.120s
I20260430 07:58:59.149843 11748 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:59.150245 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:58:59.021237 11739 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.022794 11742 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:58:59.156718 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.022046 11512 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.022337 11654 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:59.153406 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 121505us
I20260430 07:58:59.167140 9978 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:58:59.162748 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.164136 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:59.159057 11494 consensus_meta.cc:287] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.605s user 0.014s sys 0.000s
I20260430 07:58:59.168236 10007 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:58:59.076164 11626 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:59.037833 11476 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.123s
I20260430 07:58:59.149857 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:58:59.181676 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 179254us
I20260430 07:58:59.160820 11599 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:59.184132 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.183301 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:59.187144 10003 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:58:59.187347 9861 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:58:59.189157 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:59.160560 11754 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:59.211067 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.213073 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:59.214061 10007 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:59.022074 11443 consensus_queue.cc:237] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:59.219987 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:59.179198 11626 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:58:59.178872 11735 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:58:59.221864 11760 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:58:59.222329 11761 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:59.182900 11494 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:58:59.181743 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:58:59.208904 11599 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:58:59.241366 11543 consensus_meta.cc:287] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.583s user 0.005s sys 0.010s
I20260430 07:58:59.247809 11543 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.248971 11747 consensus_queue.cc:1048] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.085s
I20260430 07:58:59.251260 11543 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.247134 9931 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:58:59.232100 11626 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:59.252791 11765 consensus_queue.cc:1059] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260430 07:58:59.253229 11599 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:59.251313 9863 raft_consensus.cc:1275] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260430 07:58:59.242137 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 164261us
I20260430 07:58:59.265525 11732 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:59.268309 11751 log.cc:927] Time spent T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.458s user 0.002s sys 0.000s
I20260430 07:58:59.263280 11761 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:59.324092 11761 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:59.255934 11618 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:59.279943 11759 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:59.282416 11763 consensus_queue.cc:1048] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.123s
I20260430 07:58:59.259878 11760 raft_consensus.cc:697] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:58:59.251979 11757 log.cc:927] Time spent T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
W20260430 07:58:59.319453 11766 log.cc:927] Time spent T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.096s user 0.002s sys 0.000s
I20260430 07:58:59.363790 11759 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:59.341081 9747 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 48e81934d52a4e93be5171ceea193032 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:58:59.345492 11732 consensus_queue.cc:1048] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:58:59.218464 11754 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:59.359504 9904 connection.cc:254] RPC call timeout handler was delayed by -1.1347610189999386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19720000000000001s for a response
I20260430 07:58:59.341240 11770 raft_consensus.cc:3055] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:58:59.351248 11761 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:59.404865 11770 raft_consensus.cc:740] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:59.405123 11761 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.379570 11654 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:59.405215 11599 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.336320 10007 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:59.426685 11599 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.426450 11770 consensus_queue.cc:260] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.417511 11654 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.358497 11618 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:59.498744 11760 consensus_queue.cc:237] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.567705 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:59.568578 9863 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:57.720002 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:58:59.566249 11776 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:58:59.571156 11776 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.504987 10004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1486) took 4503 ms. Trace:
I20260430 07:58:59.572214 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:59.572854 9860 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:59.515336 9862 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:58:59.574621 11776 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.575289 11776 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:59.575572 11654 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.576102 11654 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:59.576259 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:59.576562 11654 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.576925 9863 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:58:59.429734 11761 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.411279 11765 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:59.577875 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:59.431389 11754 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:59.579568 9863 raft_consensus.cc:2393] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 4: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:58:59.580469 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:59.572527 10004 rpcz_store.cc:276] 0430 07:58:55.002002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:55.102162 (+100160us) service_pool.cc:225] Handling call
0430 07:58:55.130237 (+ 28075us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:56.110715 (+980478us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:59.504563 (+3393848us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54000492d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:58:59.504846 (+ 283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2247570688}
I20260430 07:58:59.581724 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:59.582330 9863 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:59.575311 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1394) took 3068 ms. Trace:
W20260430 07:58:59.583622 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:59.584240 11637 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:59.583041 9862 rpcz_store.cc:276] 0430 07:58:56.506586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:59.501061 (+2994475us) service_pool.cc:225] Handling call
0430 07:58:59.514608 (+ 13547us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:59.574981 (+ 60373us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:59.575257 (+ 276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:59.584786 11637 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:58:59.585470 11776 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:59.586018 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:58:59.586716 9862 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 4
I20260430 07:58:59.587682 11776 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:59.588244 11776 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:58:59.564299 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:59.590526 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:59.592615 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:59.593724 11722 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:59.593621 11565 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:58:59.594342 11722 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:59.596800 11722 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:59.598215 11722 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:59.601018 11722 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:59.603286 11722 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:59.604835 9863 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:59.606283 9903 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:59.608713 9903 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:58:59.606990 11501 consensus_queue.cc:1048] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:58:59.607116 11792 consensus_queue.cc:1059] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260430 07:58:59.599303 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:59.614032 9860 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
W20260430 07:58:59.631695 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:59.635478 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:58:59.652333 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:59.651180 9860 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:58:59.594918 11565 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:59.664562 9828 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.001s sys 0.015s
I20260430 07:58:59.427716 11770 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:59.665892 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.666796 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:59.665098 9828 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.642s user 0.002s sys 0.015s
W20260430 07:58:59.676877 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 654108us
W20260430 07:58:59.679577 9828 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.540s (SENT) [suppressed 3 similar messages]
W20260430 07:58:59.675105 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 68849.7us
I20260430 07:58:59.477252 11724 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:59.684491 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.588768 11776 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:59.519058 11759 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.690901 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:59.698792 9828 leader_election.cc:336] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.448s (SENT)
I20260430 07:58:59.698987 11776 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:59.700063 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:59.700414 9828 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:59.664662 9855 raft_consensus.cc:1217] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 3.2->[4.3-4.3] Dedup: 4.3->[]
I20260430 07:58:59.708771 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:58:59.578060 11765 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.709344 11501 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:59.717819 11501 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.718545 11773 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:58:59.727699 11765 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:59.713913 11742 consensus_meta.cc:287] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.540s user 0.011s sys 0.004s
I20260430 07:58:59.710343 11774 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:59.728097 10008 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260430 07:58:59.724895 9862 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:58:59.763010 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 155) took 2485 ms. Trace:
W20260430 07:58:59.764285 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 68848us
I20260430 07:58:59.764520 10008 rpcz_store.cc:276] 0430 07:58:57.277436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:59.727409 (+2449973us) service_pool.cc:225] Handling call
0430 07:58:59.727771 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:59.762576 (+ 34805us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:59.762943 (+ 367us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:58:59.714175 9855 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260430 07:58:59.767465 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:59.726308 11776 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:58:59.707024 11724 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:59.763629 11748 consensus_meta.cc:287] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.610s user 0.004s sys 0.010s
I20260430 07:58:59.764823 11765 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
W20260430 07:58:59.789258 9828 leader_election.cc:336] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.486s (SENT)
I20260430 07:58:59.789887 9828 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:58:59.780753 11748 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.795449 11723 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:59.787788 11501 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:59.805864 11501 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:59.785951 11565 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:59.803308 11765 raft_consensus.cc:697] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:58:59.806854 11501 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.813735 11765 consensus_queue.cc:237] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.814829 11501 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:58:59.814680 11404 consensus_queue.cc:1059] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20260430 07:58:59.819370 9828 connection.cc:254] RPC call timeout handler was delayed by -4.7027883549999956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3247s for a response
W20260430 07:58:59.820135 9828 connection.cc:254] RPC call timeout handler was delayed by -4.6218468650000091s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2271s for a response
I20260430 07:58:59.818886 11692 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260430 07:58:59.820555 9828 connection.cc:254] RPC call timeout handler was delayed by -3.9447196770000232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17930000000000001s for a response
W20260430 07:58:59.820950 9828 connection.cc:254] RPC call timeout handler was delayed by -3.9447196770000232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17930000000000001s for a response
I20260430 07:58:59.782063 11724 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:58:59.788415 11776 raft_consensus.cc:697] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:58:59.830346 9858 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1396) took 3323 ms (client timeout 1972 ms). Trace:
W20260430 07:58:59.848235 9858 rpcz_store.cc:269] 0430 07:58:56.507338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:59.829971 (+3322633us) service_pool.cc:212] Skipping call since client already timed out
0430 07:58:59.830261 (+ 290us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260430 07:58:59.849596 11724 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:58:59.849382 11776 consensus_queue.cc:237] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.850247 11724 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.762567 11774 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:58:59.771160 11742 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:58:59.830410 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 220242us
I20260430 07:58:59.796111 11723 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:58:59.840035 9828 connection.cc:254] RPC call timeout handler was delayed by -3.8448196770000322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
I20260430 07:58:59.872961 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:59.889249 10008 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:58:59.835127 11748 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:58:59.890647 9828 connection.cc:254] RPC call timeout handler was delayed by -3.8448196770000322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
W20260430 07:58:59.891300 9828 connection.cc:254] RPC call timeout handler was delayed by -3.6916305890000558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
W20260430 07:58:59.891765 9828 connection.cc:254] RPC call timeout handler was delayed by -3.6916305890000558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
W20260430 07:58:59.892385 9828 connection.cc:254] RPC call timeout handler was delayed by -3.5242305890000125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16990000000000002s for a response
W20260430 07:58:59.936265 9828 connection.cc:254] RPC call timeout handler was delayed by -3.5242305890000125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16990000000000002s for a response
W20260430 07:58:59.936662 9828 connection.cc:254] RPC call timeout handler was delayed by -3.4207305890000157s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1814s for a response
W20260430 07:58:59.937322 9828 connection.cc:254] RPC call timeout handler was delayed by -3.4207305890000157s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1814s for a response
W20260430 07:58:59.937673 9828 connection.cc:254] RPC call timeout handler was delayed by -3.3739305890000537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18660000000000002s for a response
W20260430 07:58:59.937995 9828 connection.cc:254] RPC call timeout handler was delayed by -3.3739305890000537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18660000000000002s for a response
W20260430 07:58:59.938515 9828 connection.cc:254] RPC call timeout handler was delayed by -3.3595305890000304s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1882s for a response
W20260430 07:58:59.938959 9828 connection.cc:254] RPC call timeout handler was delayed by -3.3595305890000304s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1882s for a response
W20260430 07:58:59.939698 9828 connection.cc:254] RPC call timeout handler was delayed by -3.2272305890000439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2029s for a response
W20260430 07:58:59.940080 9828 connection.cc:254] RPC call timeout handler was delayed by -3.2272305890000439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2029s for a response
W20260430 07:58:59.940611 9828 connection.cc:254] RPC call timeout handler was delayed by -3.1300305890000573s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2137s for a response
I20260430 07:58:59.936084 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:59.940991 9828 connection.cc:254] RPC call timeout handler was delayed by -3.1300305890000573s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2137s for a response
W20260430 07:58:59.945366 9828 connection.cc:254] RPC call timeout handler was delayed by -3.1300305890000573s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2137s for a response
W20260430 07:58:59.946527 9828 connection.cc:254] RPC call timeout handler was delayed by -3.1021305890000122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21680000000000002s for a response
W20260430 07:58:59.947052 9828 connection.cc:254] RPC call timeout handler was delayed by -3.1021305890000122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21680000000000002s for a response
W20260430 07:58:59.947533 9828 connection.cc:254] RPC call timeout handler was delayed by -2.9455305890000432s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23420000000000002s for a response
I20260430 07:58:59.949415 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:58:59.955090 9828 connection.cc:254] RPC call timeout handler was delayed by -2.9455305890000432s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23420000000000002s for a response
I20260430 07:58:59.958695 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.960215 10009 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
W20260430 07:58:59.958432 9828 connection.cc:254] RPC call timeout handler was delayed by -2.1193305890000147s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.326s for a response
I20260430 07:58:59.961607 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:58:59.962052 9828 connection.cc:254] RPC call timeout handler was delayed by -2.1193305890000147s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.326s for a response
W20260430 07:58:59.962430 9828 connection.cc:254] RPC call timeout handler was delayed by -2.1193305890000147s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.326s for a response
I20260430 07:58:59.963071 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.962386 10004 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:58:59.963580 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.972467 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:59.975155 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:58:59.976087 10007 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
W20260430 07:58:59.976825 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:58:59.981985 11724 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.982290 11742 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.990477 11789 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:00.018931 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:00.020046 10004 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:00.020267 11774 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:58:59.983160 11724 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:58:59.955629 11637 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.027504 11763 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:00.030599 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.038972 11792 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:00.034096 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:00.039316 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.038650 9903 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:00.055805 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:00.058043 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:58:59.991106 11789 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:00.061941 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.028568 11724 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:00.076783 11724 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:00.078974 11803 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:00.080933 11723 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:00.050810 11763 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.099851 9828 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:00.103322 11763 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:00.039551 11792 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.104568 11637 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:00.107399 11792 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:00.109459 11770 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:00.110255 11770 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:58:59.427377 11599 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:00.120354 11792 raft_consensus.cc:2804] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:59:00.122437 11792 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.123020 11792 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:00.099057 11803 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.084960 10008 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:00.112821 11599 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:00.087670 11793 log.cc:927] Time spent T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.144s user 0.001s sys 0.000s
W20260430 07:59:00.093111 10534 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.931s user 0.005s sys 0.005s
I20260430 07:59:00.086593 11789 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:00.149308 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 51731.8us
W20260430 07:59:00.152220 10534 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.991s user 0.009s sys 0.005s
W20260430 07:59:00.155248 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:00.157801 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 1.01794e+06us
W20260430 07:59:00.173349 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.211035 9828 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:00.212598 9828 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:00.214768 11797 raft_consensus.cc:2764] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260430 07:59:00.215404 9828 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:00.216533 9828 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:00.216809 11803 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:00.217512 11770 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:00.217640 11692 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:00.217924 9828 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:00.218232 11692 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:00.218613 11735 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.155043 11798 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:00.218711 11692 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:00.213725 11774 raft_consensus.cc:2764] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260430 07:59:00.207756 9858 raft_consensus.cc:2955] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:00.219219 11735 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:00.208892 11637 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:00.239944 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1302) took 1065 ms. Trace:
I20260430 07:59:00.240358 9856 rpcz_store.cc:276] 0430 07:58:59.174193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:59.174336 (+ 143us) service_pool.cc:225] Handling call
0430 07:58:59.178235 (+ 3899us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:59.582369 (+404134us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:59.582400 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:59.582915 (+ 515us) log.cc:844] Serialized 26 byte log entry
0430 07:58:59.692668 (+109753us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:58:59.692706 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:59.692781 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:00.239106 (+546325us) raft_consensus.cc:1685] finished
0430 07:59:00.239613 (+ 507us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:00.239820 (+ 207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1025,"raft.queue_time_us":31507,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":83,"thread_start_us":512032,"threads_started":2,"wal-append.queue_time_us":260260}
I20260430 07:59:00.245288 11798 consensus_queue.cc:1048] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:59:00.252905 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.257167 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.239617 11761 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:00.245033 11797 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:59:00.294821 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.266712 9828 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:00.295728 9937 raft_consensus.cc:2393] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:00.296640 9828 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:00.265758 9976 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.018s sys 0.038s
W20260430 07:59:00.310632 9976 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s user 0.018s sys 0.038s
W20260430 07:59:00.311950 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.268224 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.314035 9934 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:00.340245 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:00.298918 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:00.265486 9978 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.010s sys 0.016s
W20260430 07:59:00.296052 11760 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.000s sys 0.040s
I20260430 07:59:00.339406 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1305) took 1164 ms. Trace:
I20260430 07:59:00.343174 9931 raft_consensus.cc:2368] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
W20260430 07:59:00.341377 9978 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s user 0.010s sys 0.016s
W20260430 07:59:00.344061 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 624519us
I20260430 07:59:00.338193 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.345590 9938 raft_consensus.cc:2368] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:00.249055 11637 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.301486 11797 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:00.293622 9977 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.019s sys 0.048s
I20260430 07:59:00.348634 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:00.356606 9862 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260430 07:59:00.369323 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:00.375840 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.376320 10008 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:00.352766 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:00.366139 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 327444us
W20260430 07:59:00.343719 11760 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.001s sys 0.041s
I20260430 07:59:00.385134 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:00.385530 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 89002.9us
I20260430 07:59:00.378695 9903 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:00.386034 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.379226 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.385947 9859 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:59:00.380306 9977 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.707s user 0.021s sys 0.050s
I20260430 07:59:00.342793 9861 rpcz_store.cc:276] 0430 07:58:59.175100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:59.188356 (+ 13256us) service_pool.cc:225] Handling call
0430 07:58:59.188888 (+ 532us) raft_consensus.cc:1471] Updating replica for 3 ops
0430 07:58:59.190741 (+ 1853us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:59.190796 (+ 55us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0430 07:58:59.563748 (+372952us) log.cc:844] Serialized 78 byte log entry
0430 07:59:00.144668 (+580920us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:00.145934 (+ 1266us) log.cc:844] Serialized 14 byte log entry
0430 07:59:00.152275 (+ 6341us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:00.152377 (+ 102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:00.339102 (+186725us) raft_consensus.cc:1685] finished
0430 07:59:00.339130 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:00.339330 (+ 200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5996,"prepare.queue_time_us":734,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":30428,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":19979,"thread_start_us":456040,"threads_started":2,"wal-append.queue_time_us":616116}
I20260430 07:59:00.391242 11748 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:00.391779 11748 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.392196 11748 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:00.338728 11806 raft_consensus.cc:2764] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260430 07:59:00.399446 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.398387 9931 raft_consensus.cc:2393] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:00.404321 9862 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:00.405174 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.405205 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.407967 9862 raft_consensus.cc:2410] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:59:00.397825 9903 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:00.400233 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.408856 9937 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:00.410746 9859 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
W20260430 07:59:00.412563 9977 connection.cc:254] RPC call timeout handler was delayed by -1.9299852490000262s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
I20260430 07:59:00.413671 9828 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:00.428254 9977 connection.cc:254] RPC call timeout handler was delayed by -1.8786852490000001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1695s for a response
W20260430 07:59:00.428750 9977 connection.cc:254] RPC call timeout handler was delayed by -1.8516852490000133s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17250000000000001s for a response
W20260430 07:59:00.429533 9977 connection.cc:254] RPC call timeout handler was delayed by -1.7585997909999946s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
I20260430 07:59:00.428757 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:00.340611 11801 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:59:00.348242 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:00.431408 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.432809 9903 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:00.435320 11811 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:00.436403 11765 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.378252 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.474193 10011 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
W20260430 07:59:00.429845 9977 connection.cc:254] RPC call timeout handler was delayed by -1.7283852490000413s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18620000000000003s for a response
I20260430 07:59:00.442373 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:00.446559 10008 raft_consensus.cc:1240] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:00.476118 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.447283 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:00.476958 9931 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:00.478224 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:00.447804 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:00.479339 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:00.476148 9977 connection.cc:254] RPC call timeout handler was delayed by -1.7022852489999991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18910000000000002s for a response
W20260430 07:59:00.481356 9977 connection.cc:254] RPC call timeout handler was delayed by -1.5027692350000166s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
W20260430 07:59:00.481714 9977 connection.cc:254] RPC call timeout handler was delayed by -1.5027692350000166s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
I20260430 07:59:00.454480 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.452855 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.467657 11770 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.467531 11732 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.384557 11797 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:00.469460 11724 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:00.438591 9828 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:00.503628 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 65126.8us
I20260430 07:59:00.451243 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.481938 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.504998 9862 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:59:00.452848 11811 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:00.485692 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:00.439657 11761 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:59:00.485217 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.513039 10008 raft_consensus.cc:2410] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 6 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260430 07:59:00.452885 11810 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:00.483034 9937 raft_consensus.cc:2393] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:00.507258 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.508275 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.518791 9859 raft_consensus.cc:2393] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 4: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:00.511528 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.514263 9860 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:59:00.474241 11803 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.473683 11798 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.474455 11723 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.480855 11774 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.523537 9931 raft_consensus.cc:2410] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260430 07:59:00.482024 9977 connection.cc:254] RPC call timeout handler was delayed by -1.4962662230000205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18030000000000002s for a response
W20260430 07:59:00.534889 9977 connection.cc:254] RPC call timeout handler was delayed by -1.4007852490000232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22260000000000002s for a response
I20260430 07:59:00.530736 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:00.535386 9977 connection.cc:254] RPC call timeout handler was delayed by -1.3935852490000116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22340000000000002s for a response
I20260430 07:59:00.537397 11820 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.530699 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:00.523653 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.527015 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1309) took 1350 ms. Trace:
I20260430 07:59:00.536612 9977 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:00.484728 9929 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:00.518095 11811 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:00.525506 11801 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:59:00.539917 9859 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:00.548483 11805 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:00.549007 11814 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260430 07:59:00.547961 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.554629 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1429) took 1206 ms. Trace:
I20260430 07:59:00.549702 9860 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:59:00.501612 11770 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:00.555841 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.555116 9857 rpcz_store.cc:276] 0430 07:58:59.348247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:59.650734 (+302487us) service_pool.cc:225] Handling call
0430 07:58:59.662348 (+ 11614us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:00.127565 (+465217us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:00.127584 (+ 19us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:00.139606 (+ 12022us) log.cc:844] Serialized 26 byte log entry
0430 07:59:00.452108 (+312502us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:00.452146 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:00.452214 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:00.554336 (+102122us) raft_consensus.cc:1685] finished
0430 07:59:00.554360 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:00.554566 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2025,"raft.queue_time_us":55519,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"thread_start_us":720544,"threads_started":2,"wal-append.queue_time_us":403226}
I20260430 07:59:00.560024 9860 raft_consensus.cc:2410] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:59:00.518225 11765 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:00.543741 9854 rpcz_store.cc:276] 0430 07:58:59.176262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:59.593813 (+417551us) service_pool.cc:225] Handling call
0430 07:58:59.598895 (+ 5082us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:58:59.828999 (+230104us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:59.829051 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:58:59.832952 (+ 3901us) log.cc:844] Serialized 52 byte log entry
0430 07:59:00.317863 (+484911us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:00.317928 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:00.318040 (+ 112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:00.526690 (+208650us) raft_consensus.cc:1685] finished
0430 07:59:00.526722 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:00.526936 (+ 214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54321,"raft.queue_time_us":103418,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":9259,"thread_start_us":660821,"threads_started":2,"wal-append.queue_time_us":633117}
I20260430 07:59:00.510133 11732 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:00.539279 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.562114 9861 raft_consensus.cc:2393] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:00.562206 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.568991 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:00.527019 11810 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.570940 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:00.571280 11810 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:00.574273 11827 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:00.574839 11827 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.575207 11827 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:00.546953 11724 raft_consensus.cc:697] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:00.563876 11818 consensus_queue.cc:1059] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20260430 07:59:00.543514 11774 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:00.587666 9854 raft_consensus.cc:1217] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 3.2->[5.3-5.3] Dedup: 5.3->[]
I20260430 07:59:00.599776 9854 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260430 07:59:00.594188 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.534683 11803 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:00.586967 11802 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:00.576102 11765 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:00.545902 11723 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:00.608333 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.605841 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:00.609824 9862 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:00.606491 11812 log.cc:927] Time spent T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.197s user 0.000s sys 0.002s
I20260430 07:59:00.561290 11729 consensus_queue.cc:1059] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20260430 07:59:00.534412 11806 consensus_queue.cc:1048] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260430 07:59:00.538625 11807 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:00.597936 11637 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:00.549170 11805 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:00.615608 10004 raft_consensus.cc:1275] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:00.602155 11789 raft_consensus.cc:3055] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:00.546706 11798 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:00.640857 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1310) took 1460 ms. Trace:
I20260430 07:59:00.641357 9863 rpcz_store.cc:276] 0430 07:58:59.180425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:59.646153 (+465728us) service_pool.cc:225] Handling call
0430 07:58:59.646549 (+ 396us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:59.818462 (+171913us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:58:59.823547 (+ 5085us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:59.824223 (+ 676us) log.cc:844] Serialized 26 byte log entry
0430 07:59:00.266117 (+441894us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:00.266176 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:00.266283 (+ 107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:00.640621 (+374338us) raft_consensus.cc:1685] finished
0430 07:59:00.640641 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:00.640794 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6299,"raft.queue_time_us":55244,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":3946,"thread_start_us":612620,"threads_started":2,"wal-append.queue_time_us":522328}
W20260430 07:59:00.645623 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.621985 11729 consensus_queue.cc:1243] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer 48e81934d52a4e93be5171ceea193032 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:59:00.622877 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:00.541870 11820 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:00.625566 11789 raft_consensus.cc:740] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:00.660799 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:00.657819 11796 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:00.619844 11802 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:00.673453 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.675896 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:00.676980 10008 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:00.677455 9862 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:00.683717 11802 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:00.624117 11807 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:00.693485 11789 consensus_queue.cc:260] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:00.724434 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:00.725180 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.727628 10004 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:59:00.677256 9930 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:00.659332 11724 consensus_queue.cc:237] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.769825 9859 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:00.771234 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:00.719835 11732 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:00.721725 10033 debug-util.cc:398] Leaking SignalData structure 0x7b080017abe0 after lost signal to thread 8673
I20260430 07:59:00.732676 11801 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:00.782467 9931 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:00.783178 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1518) took 3049 ms. Trace:
I20260430 07:59:00.732764 11789 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:00.787714 10005 rpcz_store.cc:276] 0430 07:58:57.724859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:57.724999 (+ 140us) service_pool.cc:225] Handling call
0430 07:58:57.725386 (+ 387us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:58:57.994007 (+268621us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:00.773482 (+2779475us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b540018e2d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:00.773685 (+ 203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522837888}
I20260430 07:59:00.788295 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:00.799108 11805 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:00.772010 11798 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:00.789088 9931 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:00.786676 11801 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.825266 11765 consensus_queue.cc:1059] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:00.826637 11765 raft_consensus.cc:3055] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:00.827037 11765 raft_consensus.cc:740] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:59:00.830468 11828 log.cc:927] Time spent T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260430 07:59:00.828073 11765 consensus_queue.cc:260] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.832306 11765 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:00.816512 11819 log.cc:927] Time spent T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20260430 07:59:00.799219 11803 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:00.801776 11816 log.cc:927] Time spent T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.168s user 0.001s sys 0.001s
I20260430 07:59:00.795990 11820 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:00.839062 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:00.853446 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.873679 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:00.874055 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 498258us
I20260430 07:59:00.874652 10007 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:59:00.838917 11774 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:00.887804 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.895483 9904 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:00.842922 11807 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:00.876364 11770 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:00.909039 9904 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:00.888052 9932 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.015s sys 0.031s
I20260430 07:59:00.910719 9904 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:00.907819 11820 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:00.918442 9904 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:00.914796 9932 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.015s sys 0.031s
W20260430 07:59:00.926190 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 316223us
I20260430 07:59:00.909448 11796 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:00.916723 11792 consensus_meta.cc:287] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.792s user 0.019s sys 0.001s
I20260430 07:59:00.918047 11789 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:00.932906 9904 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:00.917486 11820 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:00.976348 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 43551.3us
W20260430 07:59:00.935926 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 431382us
I20260430 07:59:00.939661 10005 raft_consensus.cc:1275] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
W20260430 07:59:00.977944 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:00.978272 10005 pending_rounds.cc:85] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 2
I20260430 07:59:00.979132 10005 pending_rounds.cc:107] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260430 07:59:00.978969 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:00.979521 10005 raft_consensus.cc:2889] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:00.926028 11803 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:00.977142 9904 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:00.923985 11770 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:59:00.990262 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:00.924523 11811 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.926600 11723 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:00.994163 11741 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:01.010665 9931 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:01.003103 11741 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:00.975790 11820 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:01.019909 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:01.021174 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:01.027048 9978 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.029s sys 0.027s
I20260430 07:59:00.995218 11774 raft_consensus.cc:2764] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260430 07:59:00.967798 11827 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:01.048038 10559 raft_consensus.cc:1217] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Deduplicated request from leader. Original: 1.4->[2.5-2.5] Dedup: 2.5->[]
I20260430 07:59:01.069758 10559 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260430 07:59:01.071065 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.050104 11811 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:01.049134 11810 consensus_queue.cc:1059] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260430 07:59:00.998245 11836 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:01.005198 9828 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:01.095109 11811 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:01.095827 11811 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:00.982860 11818 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:00.936184 11792 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:01.061061 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:01.105544 10005 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:00.992390 11805 consensus_queue.cc:1048] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
W20260430 07:59:01.054190 11830 log.cc:927] Time spent T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.187s user 0.000s sys 0.004s
W20260430 07:59:01.068130 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:01.068054 9978 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:01.005787 11770 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260430 07:59:01.117522 11841 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260430 07:59:01.099431 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:01.118600 10005 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:01.119100 9931 raft_consensus.cc:2393] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:01.056411 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 79311.1us
I20260430 07:59:01.119920 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:01.120609 9904 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:01.123147 11811 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:01.102866 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 97699.4us
I20260430 07:59:01.105647 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:01.126246 9854 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:01.127689 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:01.127897 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 59931.7us
I20260430 07:59:01.106843 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:01.128388 9856 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:59:01.086068 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.129484 11637 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:01.129916 11637 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:01.130318 11637 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:00.935725 11796 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:01.131356 9828 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:01.135475 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:01.137755 9856 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:59:01.131450 9978 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:01.145560 9828 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:01.149890 9904 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:01.101549 11741 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:01.175778 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:01.181113 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.181875 11836 raft_consensus.cc:2764] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260430 07:59:01.182729 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.184461 9904 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:01.186091 9904 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:01.188174 9904 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:01.188953 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:01.143801 11512 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:01.202611 9828 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:01.205652 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.208171 11807 raft_consensus.cc:3055] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:01.212198 9828 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:01.187932 11811 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:01.198796 11770 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:01.208668 11807 raft_consensus.cc:740] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:59:01.229638 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 113550us
I20260430 07:59:01.230564 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.204844 11724 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:01.209465 11803 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:59:01.246448 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 213727us
I20260430 07:59:01.250721 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:01.252981 11512 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:01.224205 11765 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:01.241071 9935 raft_consensus.cc:2393] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:01.208949 11836 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:01.221890 11841 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:01.232810 9903 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:01.223733 11774 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:01.230342 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:01.343538 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 92920.6us
W20260430 07:59:01.338085 11626 consensus_meta.cc:287] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 2.049s user 0.015s sys 0.002s
I20260430 07:59:01.345472 9828 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:01.345398 11626 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:01.347260 10010 raft_consensus.cc:2368] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
W20260430 07:59:01.340068 11748 consensus_meta.cc:287] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.947s user 0.004s sys 0.009s
I20260430 07:59:01.314641 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.352104 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:01.256573 11692 consensus_meta.cc:287] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 1.035s user 0.010s sys 0.005s
W20260430 07:59:01.305450 11797 consensus_meta.cc:287] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.791s user 0.007s sys 0.007s
W20260430 07:59:01.314124 11833 log.cc:927] Time spent T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.271s user 0.000s sys 0.001s
W20260430 07:59:01.325551 10033 debug-util.cc:398] Leaking SignalData structure 0x7b08000ebea0 after lost signal to thread 9795
W20260430 07:59:01.266412 11735 consensus_meta.cc:287] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 1.022s user 0.011s sys 0.005s
W20260430 07:59:01.290058 11843 log.cc:927] Time spent T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.236s user 0.001s sys 0.001s
I20260430 07:59:01.339380 11619 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:01.371125 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.323110 11796 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:01.325263 11792 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:01.383901 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 151141us
I20260430 07:59:01.338658 11827 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:01.348094 11765 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:01.348273 11836 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:01.367354 11654 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:01.399361 9747 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 48e81934d52a4e93be5171ceea193032 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
W20260430 07:59:01.402366 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 270933us
I20260430 07:59:01.322289 11637 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:01.369897 11512 raft_consensus.cc:2749] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:01.394124 11811 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:01.389307 11841 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:01.422205 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.411229 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:01.423127 10009 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:01.365545 11748 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:01.422399 9863 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:01.425334 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:01.415967 9903 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:59:01.410989 9828 connection.cc:254] RPC call timeout handler was delayed by -1.0336420869999756s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
W20260430 07:59:01.428568 9828 connection.cc:254] RPC call timeout handler was delayed by -1.0246420870000179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
I20260430 07:59:01.429672 9828 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:01.285257 11807 consensus_queue.cc:260] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:01.408185 11765 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:01.404958 11619 raft_consensus.cc:697] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:01.393893 11735 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:01.408365 11836 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:01.464713 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 48807.3us
I20260430 07:59:01.482996 10003 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:01.369181 11797 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:01.488668 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 58930.6us
I20260430 07:59:01.489799 9828 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:01.491238 11512 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:01.491514 9828 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:01.493741 9937 raft_consensus.cc:2393] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:01.426795 11841 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:01.492220 11748 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:01.368906 11692 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:01.517023 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:01.437719 11798 raft_consensus.cc:2749] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:01.438655 11811 raft_consensus.cc:697] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:01.437928 11827 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260430 07:59:01.528370 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:01.485634 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.504269 11840 consensus_queue.cc:1048] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
I20260430 07:59:01.501806 11792 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:01.542294 9858 raft_consensus.cc:1240] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:01.551360 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:01.552420 10011 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:01.547940 11789 consensus_meta.cc:287] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.578s user 0.006s sys 0.006s
W20260430 07:59:01.571542 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:01.541751 11807 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:01.541065 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 173) took 4371 ms. Trace:
I20260430 07:59:01.546041 11626 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:01.549541 11774 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:01.564767 11618 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:01.515619 11724 consensus_queue.cc:1059] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260430 07:59:01.586592 11789 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:59:01.610862 11845 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
W20260430 07:59:01.615751 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:01.631265 10009 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:59:01.647892 9903 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:01.649027 11724 raft_consensus.cc:2804] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:59:01.649777 11724 raft_consensus.cc:697] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:01.651386 11724 consensus_queue.cc:237] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:01.607093 11797 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:01.570930 11619 consensus_queue.cc:237] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:01.655866 10009 raft_consensus.cc:1275] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260430 07:59:01.659520 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 232327us
I20260430 07:59:01.652902 11637 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:01.664549 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.606925 11741 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:01.615176 11811 consensus_queue.cc:237] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:01.819290 10011 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:01.943106 11806 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:01.786578 11765 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:01.688189 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:01.988737 11841 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:01.990860 11818 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:01.991540 11841 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:01.895907 11640 consensus_meta.cc:287] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 2.933s user 0.016s sys 0.001s
I20260430 07:59:01.901728 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1489) took 6464 ms. Trace:
W20260430 07:59:01.738852 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 488648us
I20260430 07:59:01.856246 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:01.960269 9856 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.006s sys 0.018s
I20260430 07:59:01.993947 9934 raft_consensus.cc:2393] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:01.994138 9856 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s user 0.007s sys 0.018s
I20260430 07:59:01.970304 9931 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:59:01.995677 9904 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.807s user 0.035s sys 0.098s
W20260430 07:59:01.996291 9904 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.807s user 0.035s sys 0.098s
W20260430 07:59:01.996970 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 808840us
W20260430 07:59:01.971212 11826 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.006s sys 0.000s
I20260430 07:59:01.669191 11748 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.306s
I20260430 07:59:01.592577 9932 rpcz_store.cc:276] 0430 07:58:57.158968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.119453 (+2960485us) service_pool.cc:225] Handling call
0430 07:59:00.145922 (+ 26469us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:00.239071 (+ 93149us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:00.245109 (+ 6038us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:00.254600 (+ 9491us) log.cc:844] Serialized 26 byte log entry
0430 07:59:00.931029 (+676429us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:00.936046 (+ 5017us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:00.969855 (+ 33809us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:01.505248 (+535393us) raft_consensus.cc:1685] finished
0430 07:59:01.505300 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:01.530182 (+ 24882us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31635,"raft.queue_time_us":33753,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":1363,"thread_start_us":667945,"threads_started":1,"wal-append.queue_time_us":712716}
W20260430 07:59:01.924340 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 216247us
I20260430 07:59:01.999063 9904 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:01.998975 9934 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 896) took 1698 ms (client timeout 1969 ms). Trace:
I20260430 07:59:01.680271 11692 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:01.677583 11512 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:01.878966 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1260) took 1305 ms. Trace:
I20260430 07:59:01.737498 11735 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:01.923336 10007 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.009s sys 0.013s
W20260430 07:59:02.013231 10007 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.009s sys 0.017s
I20260430 07:59:02.005307 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:01.887809 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:02.018254 11774 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:02.018795 10563 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54754 (request call id 246) took 1119 ms. Trace:
I20260430 07:59:02.019559 10563 rpcz_store.cc:276] 0430 07:59:00.899561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:01.882079 (+982518us) service_pool.cc:225] Handling call
0430 07:59:02.018622 (+136543us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:02.003762 9904 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:01.857837 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:02.027976 10011 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
W20260430 07:59:01.996979 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:01.978243 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:01.972864 11806 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:01.984732 11741 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:01.985628 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 493979us
W20260430 07:59:01.901295 10534 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.038s sys 0.094s
W20260430 07:59:02.158169 10534 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.908s user 0.038s sys 0.094s
W20260430 07:59:02.158684 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 908569us
W20260430 07:59:01.964450 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 298863us
I20260430 07:59:01.991426 11818 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:01.874601 11571 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260430 07:59:01.993139 10012 rpcz_store.cc:276] 0430 07:58:55.437112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:55.437252 (+ 140us) service_pool.cc:225] Handling call
0430 07:58:55.437791 (+ 539us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:55.450675 (+ 12884us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:55.450740 (+ 65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:58:55.451401 (+ 661us) log.cc:844] Serialized 26 byte log entry
0430 07:58:55.505428 (+ 54027us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:58:55.505463 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:58:55.505530 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:58:57.284739 (+1779209us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:01.901229 (+4616490us) raft_consensus.cc:1685] finished
0430 07:59:01.901288 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:01.901630 (+ 342us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":85756,"raft.queue_time_us":32496,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":13150,"spinlock_wait_cycles":28580096,"thread_start_us":31746,"threads_started":1,"wal-append.queue_time_us":167116,"wal-append.run_cpu_time_us":16214,"wal-append.run_wall_time_us":3562734}
I20260430 07:59:02.171908 11765 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:02.135385 11774 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:01.997937 11640 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:01.768846 11847 log.cc:927] Time spent T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
W20260430 07:59:01.779584 11852 log.cc:927] Time spent T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.141s user 0.002s sys 0.000s
W20260430 07:59:01.758737 11820 consensus_meta.cc:287] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.682s user 0.013s sys 0.001s
I20260430 07:59:02.004617 9903 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:02.013257 9937 rpcz_store.cc:276] 0430 07:59:00.573479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.576911 (+ 3432us) service_pool.cc:225] Handling call
0430 07:59:01.878613 (+1301702us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b540036d8d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:01.878825 (+ 212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2051725056}
W20260430 07:59:02.002732 9934 rpcz_store.cc:269] 0430 07:59:00.300733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.338456 (+ 37723us) service_pool.cc:225] Handling call
0430 07:59:01.856012 (+1517556us) spinlock_profiling.cc:243] Waited 3.69 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:01.998910 (+142898us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8495488}
W20260430 07:59:01.998504 11810 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.004s sys 0.005s
I20260430 07:59:02.006100 11792 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:59:02.040618 11826 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.008s sys 0.000s
W20260430 07:59:02.180584 9860 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.014s sys 0.033s
W20260430 07:59:02.235349 9860 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.787s user 0.015s sys 0.034s
W20260430 07:59:02.184955 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 190992us
I20260430 07:59:02.176457 11469 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:02.238230 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 77225.3us
I20260430 07:59:02.238433 11469 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:02.207105 11637 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:02.208968 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:02.209957 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:02.217401 9936 raft_consensus.cc:2410] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260430 07:59:01.968436 11836 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:02.262377 10008 raft_consensus.cc:2393] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:02.359519 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:02.361066 9937 raft_consensus.cc:2410] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:02.359426 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:02.367112 10009 raft_consensus.cc:2368] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4.
W20260430 07:59:02.388090 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1263) took 1803 ms (client timeout 1742 ms). Trace:
W20260430 07:59:02.388624 9937 rpcz_store.cc:269] 0430 07:59:00.585121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:02.208612 (+1623491us) service_pool.cc:225] Handling call
0430 07:59:02.388007 (+179395us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:02.390618 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:02.392130 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:02.295085 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 55690us
W20260430 07:59:02.286967 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 49407.6us
I20260430 07:59:02.397866 9977 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:02.217243 11810 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.775s user 0.005s sys 0.006s
W20260430 07:59:02.395602 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 391729us
I20260430 07:59:02.406275 9904 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:02.424011 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:02.286403 11692 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:02.387012 11841 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:02.425429 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:02.433107 11692 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:02.436198 11640 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:02.440281 11692 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:02.436415 11841 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:02.431114 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 71681.7us
I20260430 07:59:02.414177 9978 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:02.445463 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1344) took 1033 ms. Trace:
I20260430 07:59:02.446923 11640 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260430 07:59:02.448287 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 50478.1us
I20260430 07:59:02.436764 11818 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:02.453626 11841 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:02.235745 11820 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:02.455559 9978 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:02.446000 9860 rpcz_store.cc:276] 0430 07:59:01.411860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:01.412031 (+ 171us) service_pool.cc:225] Handling call
0430 07:59:01.447300 (+ 35269us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:01.447586 (+ 286us) raft_consensus.cc:1534] Early marking committed up to index 4
0430 07:59:02.236344 (+788758us) log.cc:844] Serialized 14 byte log entry
0430 07:59:02.425370 (+189026us) log.cc:844] Serialized 14 byte log entry
0430 07:59:02.444988 (+ 19618us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:02.445061 (+ 73us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:02.445125 (+ 64us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:02.445212 (+ 87us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:02.445385 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49808,"prepare.queue_time_us":75544,"prepare.run_cpu_time_us":142,"prepare.run_wall_time_us":6423,"raft.queue_time_us":71365,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":328,"thread_start_us":936323,"threads_started":2,"wal-append.queue_time_us":148855}
I20260430 07:59:02.748497 9828 leader_election.cc:400] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:02.876909 9828 leader_election.cc:403] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260430 07:59:02.216567 11774 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:02.448633 10006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1499) took 6450 ms. Trace:
I20260430 07:59:02.883204 10006 rpcz_store.cc:276] 0430 07:58:55.998140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:56.000889 (+ 2749us) service_pool.cc:225] Handling call
0430 07:58:56.001397 (+ 508us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:56.001863 (+ 466us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:58:56.001902 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:01.898339 (+5896437us) spinlock_profiling.cc:243] Waited 1.08 ms on lock 0x7b2c0008b928. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb567b7dc81 00007fb567b8d5e2 00007fb5692bede1 00007fb56928381a 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22
0430 07:59:01.998672 (+100333us) log.cc:844] Serialized 26 byte log entry
0430 07:59:02.176221 (+177549us) raft_consensus.cc:1645] Marking committed up to 1
0430 07:59:02.176287 (+ 66us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:02.176657 (+ 370us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:02.448298 (+271641us) raft_consensus.cc:1685] finished
0430 07:59:02.448327 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:02.448557 (+ 230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8521,"raft.queue_time_us":1918,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":19899,"spinlock_wait_cycles":2488576,"thread_start_us":169940,"threads_started":1,"wal-append.queue_time_us":440302}
W20260430 07:59:02.912535 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 506109us
I20260430 07:59:02.888729 11512 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:02.919543 11853 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:02.920068 11853 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:02.919857 11512 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:02.921886 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:02.920892 11774 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:02.436715 9937 raft_consensus.cc:2410] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:02.304342 11850 consensus_queue.cc:1048] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.571s
I20260430 07:59:02.425987 11864 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260430 07:59:02.969209 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 513489us
W20260430 07:59:02.968849 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1270) took 2364 ms (client timeout 1803 ms). Trace:
W20260430 07:59:02.971329 9937 rpcz_store.cc:269] 0430 07:59:00.605996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:02.404356 (+1798360us) service_pool.cc:225] Handling call
0430 07:59:02.968740 (+564384us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:02.879777 11640 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260430 07:59:02.976667 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1264) took 2391 ms (client timeout 1863 ms). Trace:
W20260430 07:59:02.977373 9937 rpcz_store.cc:269] 0430 07:59:00.585497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:02.976383 (+2390886us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:02.976579 (+ 196us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260430 07:59:02.880422 11663 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:02.889428 11841 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:03.003124 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 86358.6us
I20260430 07:59:03.003895 9904 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:03.005915 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 1.00091e+06us
W20260430 07:59:03.007400 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1265) took 2392 ms (client timeout 1871 ms). Trace:
W20260430 07:59:03.008033 9937 rpcz_store.cc:269] 0430 07:59:00.585831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:02.978351 (+2392520us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:02.978502 (+ 151us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260430 07:59:03.004696 9977 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:02.909087 11692 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:03.005226 9904 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:02.973474 11724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.001s sys 0.009s
W20260430 07:59:02.976016 11877 log.cc:927] Time spent T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20260430 07:59:03.027810 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1286) took 2372 ms (client timeout 1863 ms). Trace:
I20260430 07:59:02.448633 11806 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:03.028823 9937 rpcz_store.cc:269] 0430 07:59:00.642793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.015269 (+2372476us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:03.015462 (+ 193us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260430 07:59:03.034089 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:03.038017 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1262) took 2453 ms (client timeout 1994 ms). Trace:
I20260430 07:59:02.975554 11859 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:02.972666 11774 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:03.027522 11724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s user 0.001s sys 0.009s
I20260430 07:59:03.016119 11818 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:02.975694 11640 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:03.038551 9937 rpcz_store.cc:269] 0430 07:59:00.584790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.037788 (+2452998us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:03.037945 (+ 157us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260430 07:59:03.045383 11860 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:03.021068 11864 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:03.051698 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:03.058018 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1261) took 2479 ms (client timeout 2068 ms). Trace:
W20260430 07:59:03.060304 9937 rpcz_store.cc:269] 0430 07:59:00.578714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.057787 (+2479073us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:03.057942 (+ 155us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260430 07:59:03.042914 11663 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election won for term 6
W20260430 07:59:03.061443 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1278) took 2429 ms (client timeout 2066 ms). Trace:
I20260430 07:59:03.034962 11735 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:03.066692 9937 rpcz_store.cc:269] 0430 07:59:00.632550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.061078 (+2428528us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:03.061222 (+ 144us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260430 07:59:03.070842 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1288) took 2427 ms (client timeout 2067 ms). Trace:
W20260430 07:59:03.072218 9937 rpcz_store.cc:269] 0430 07:59:00.643767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.070616 (+2426849us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:03.070770 (+ 154us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260430 07:59:03.079303 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1289) took 2435 ms (client timeout 2069 ms). Trace:
W20260430 07:59:03.079994 9937 rpcz_store.cc:269] 0430 07:59:00.644091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.079056 (+2434965us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:03.079222 (+ 166us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260430 07:59:03.044091 11859 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:03.020319 11841 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:03.081229 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1267) took 2485 ms (client timeout 2199 ms). Trace:
W20260430 07:59:03.048991 10560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54754 (request call id 242) took 2236 ms (client timeout 1655 ms). Trace:
I20260430 07:59:03.058295 11640 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:03.028623 11692 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:03.085533 9937 rpcz_store.cc:269] 0430 07:59:00.595761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.081010 (+2485249us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:03.081158 (+ 148us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260430 07:59:03.035877 10534 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.018s sys 0.034s
W20260430 07:59:03.078329 10532 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.000s sys 0.023s
I20260430 07:59:03.052841 11774 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:03.088552 9901 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.952s user 0.003s sys 0.019s
W20260430 07:59:03.176942 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 171716us
W20260430 07:59:03.146665 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 702467us
W20260430 07:59:03.176393 10534 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.816s user 0.019s sys 0.036s
I20260430 07:59:03.064079 11806 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:03.127734 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:03.069638 11818 raft_consensus.cc:2749] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:03.177758 9901 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.131s user 0.003s sys 0.019s
W20260430 07:59:03.194692 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 1.03375e+06us
W20260430 07:59:03.172927 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 202277us
W20260430 07:59:03.081990 11826 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.028s sys 0.102s
I20260430 07:59:03.193984 9904 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:59:03.196311 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 191640us
I20260430 07:59:03.085168 11859 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:03.195914 9828 leader_election.cc:400] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:03.202638 9828 leader_election.cc:403] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260430 07:59:03.176393 10532 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.786s user 0.000s sys 0.024s
W20260430 07:59:03.166724 11874 log.cc:927] Time spent T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260430 07:59:03.130081 11512 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:03.208283 10531 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.007s sys 0.012s
I20260430 07:59:03.197379 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:03.209344 11801 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:03.209515 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1284) took 2567 ms (client timeout 2901 ms). Trace:
W20260430 07:59:03.210181 9937 rpcz_store.cc:269] 0430 07:59:00.642237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.176209 (+2533972us) service_pool.cc:225] Handling call
0430 07:59:03.209398 (+ 33189us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:03.211225 10561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54754 (request call id 245) took 2334 ms (client timeout 1743 ms). Trace:
I20260430 07:59:03.196789 9933 raft_consensus.cc:2393] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:03.205806 11884 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:03.176956 11836 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:03.208655 10531 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.809s user 0.007s sys 0.012s
W20260430 07:59:03.211328 10033 debug-util.cc:398] Leaking SignalData structure 0x7b0800385b40 after lost signal to thread 9887
I20260430 07:59:03.082959 11810 raft_consensus.cc:2749] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:03.165349 11858 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:03.081905 11663 raft_consensus.cc:697] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:59:03.165162 11870 log.cc:927] Time spent T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.581s user 0.002s sys 0.000s
I20260430 07:59:03.176792 11820 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:03.212890 11774 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:03.157001 11854 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:03.221004 9937 raft_consensus.cc:1275] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:03.115909 11853 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:03.201594 11826 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.873s user 0.032s sys 0.108s
I20260430 07:59:03.066242 11864 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.243350 11663 consensus_queue.cc:237] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:03.260577 11853 consensus_queue.cc:1059] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260430 07:59:03.211730 10561 rpcz_store.cc:269] 0430 07:59:00.876846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.210885 (+2334039us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:03.211011 (+ 126us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260430 07:59:03.082141 11735 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.231894 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:03.234364 10012 raft_consensus.cc:1240] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260430 07:59:03.266849 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:03.267581 9863 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:03.268419 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:03.082635 11860 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:03.235864 9933 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 893) took 2924 ms (client timeout 1612 ms). Trace:
W20260430 07:59:03.083832 10560 rpcz_store.cc:269] 0430 07:59:00.810026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.034618 (+2224592us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:03.046043 (+ 11425us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260430 07:59:03.269560 9933 rpcz_store.cc:269] 0430 07:59:00.299375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.301699 (+ 2324us) service_pool.cc:225] Handling call
0430 07:59:03.127527 (+2825828us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:03.223300 (+ 95773us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4625792}
I20260430 07:59:03.273147 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:03.273738 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:03.242367 11858 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:03.214290 11806 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:03.281744 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:03.241504 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 46146.6us
I20260430 07:59:03.247735 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:03.284912 9862 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
W20260430 07:59:03.222645 10033 debug-util.cc:398] Leaking SignalData structure 0x7b08003281a0 after lost signal to thread 9888
I20260430 07:59:03.251613 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:03.286005 10003 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:03.252480 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:03.218680 11818 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:03.223506 11811 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:03.302356 9860 raft_consensus.cc:2393] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:03.281750 11774 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:03.212663 11884 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:03.259936 10005 raft_consensus.cc:1217] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Deduplicated request from leader. Original: 3.2->[5.3-5.3] Dedup: 5.3->[]
I20260430 07:59:03.323236 10005 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260430 07:59:03.277868 11853 raft_consensus.cc:3055] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:03.276865 11841 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:03.330849 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:03.245450 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1266) took 2659 ms. Trace:
I20260430 07:59:03.332453 9937 rpcz_store.cc:276] 0430 07:59:00.592811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.211627 (+2618816us) service_pool.cc:225] Handling call
0430 07:59:03.213742 (+ 2115us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:03.245135 (+ 31393us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:03.245359 (+ 224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:03.261359 11854 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.290838 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:03.337158 9862 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:03.224812 11876 consensus_queue.cc:1059] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260430 07:59:03.318835 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:03.343227 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:03.276007 11692 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:03.334295 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:03.282613 11858 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:03.325712 11853 raft_consensus.cc:740] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:03.332175 11774 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:03.362974 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:03.363994 10003 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:03.357185 9937 raft_consensus.cc:1275] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:03.367280 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1273) took 2759 ms. Trace:
I20260430 07:59:03.367811 9937 rpcz_store.cc:276] 0430 07:59:00.608166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.333737 (+2725571us) service_pool.cc:225] Handling call
0430 07:59:03.356755 (+ 23018us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:03.365255 (+ 8500us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:03.367200 (+ 1945us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:03.320546 11818 raft_consensus.cc:697] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:03.381372 9937 raft_consensus.cc:3055] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:03.381805 9937 raft_consensus.cc:740] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:59:03.368674 10033 debug-util.cc:398] Leaking SignalData structure 0x7b0800174940 after lost signal to thread 9962
I20260430 07:59:03.385859 9862 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:03.386960 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:03.387130 9978 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:03.376781 11864 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:03.387578 9862 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:59:03.400570 9863 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:03.406924 9978 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:03.408464 10012 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:03.317725 11811 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.405670 9937 consensus_queue.cc:260] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.412047 11890 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:03.415401 9937 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:03.417496 9862 raft_consensus.cc:1275] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:03.419210 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:03.420181 9862 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:03.421216 9862 raft_consensus.cc:1240] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
W20260430 07:59:03.404876 10033 debug-util.cc:398] Leaking SignalData structure 0x7b08002f66c0 after lost signal to thread 9963
I20260430 07:59:03.424943 9978 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:03.430792 11443 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 4.205s user 0.029s sys 0.034s
I20260430 07:59:03.435792 9862 raft_consensus.cc:1275] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260430 07:59:03.439569 11811 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:03.436380 11853 consensus_queue.cc:260] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.441262 11853 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:03.443959 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:03.442005 9930 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1269) took 2838 ms (client timeout 1627 ms). Trace:
I20260430 07:59:03.451350 9863 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:03.423086 11810 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:03.456051 9862 raft_consensus.cc:1240] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:03.455287 11810 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:03.459261 11810 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:03.454710 10565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54770 (request call id 311) took 2349 ms (client timeout 2604 ms). Trace:
W20260430 07:59:03.450618 9930 rpcz_store.cc:269] 0430 07:59:00.603867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.607442 (+ 3575us) service_pool.cc:225] Handling call
0430 07:59:03.441845 (+2834403us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b54002f1554. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:03.441937 (+ 92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25863552}
W20260430 07:59:03.465821 10565 rpcz_store.cc:269] 0430 07:59:01.100936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:01.423539 (+322603us) service_pool.cc:225] Handling call
0430 07:59:03.443824 (+2020285us) spinlock_profiling.cc:243] Waited 38.9 ms on lock 0x7b58001f1798. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:03.450554 (+ 6730us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":89388288}
I20260430 07:59:03.472826 11889 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:59:03.475914 9747 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 48e81934d52a4e93be5171ceea193032 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:03.478152 11735 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:03.478638 11803 raft_consensus.cc:3055] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:03.479297 11803 raft_consensus.cc:740] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:03.444203 11640 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.443284 11890 raft_consensus.cc:697] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:03.482229 11640 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:03.437368 11854 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:03.473488 11889 raft_consensus.cc:697] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:03.470719 9930 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:03.498345 11884 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:03.442719 11818 consensus_queue.cc:237] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.472648 11767 raft_consensus.cc:2955] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:03.489569 11774 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:03.519801 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1276) took 2904 ms. Trace:
I20260430 07:59:03.520375 9930 rpcz_store.cc:276] 0430 07:59:00.627669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.470008 (+2842339us) service_pool.cc:225] Handling call
0430 07:59:03.470448 (+ 440us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:03.509310 (+ 38862us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:03.519313 (+ 10003us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:03.520072 11889 consensus_queue.cc:237] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.526173 9930 raft_consensus.cc:1275] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:59:03.481240 11803 consensus_queue.cc:260] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:03.526254 11869 log.cc:927] Time spent T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 1.527s user 0.025s sys 0.038s
W20260430 07:59:03.530997 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 330398us
W20260430 07:59:03.541880 11443 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 4.319s user 0.030s sys 0.034s
W20260430 07:59:03.552561 11776 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 3.572s user 0.000s sys 0.016s
W20260430 07:59:03.553046 11776 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.572s user 0.000s sys 0.017s
I20260430 07:59:03.556610 11774 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:03.557473 11774 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.560173 11774 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:03.535219 11730 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:03.582567 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 344011us
I20260430 07:59:03.564875 11803 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:03.589499 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:03.551491 11860 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:03.564990 11443 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:03.593636 9903 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.009s sys 0.007s
I20260430 07:59:03.569825 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1279) took 2931 ms. Trace:
W20260430 07:59:03.599251 9903 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.555s user 0.009s sys 0.007s
W20260430 07:59:03.602854 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 559804us
I20260430 07:59:03.600144 9930 rpcz_store.cc:276] 0430 07:59:00.639972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.525331 (+2885359us) service_pool.cc:225] Handling call
0430 07:59:03.525742 (+ 411us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:03.535492 (+ 9750us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:03.565437 (+ 29945us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:03.618747 9937 raft_consensus.cc:1275] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:03.623371 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:03.625747 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1275) took 3011 ms. Trace:
W20260430 07:59:03.624434 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 397192us
W20260430 07:59:03.633112 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 196829us
I20260430 07:59:03.602934 11443 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:03.622404 11776 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:03.618669 11895 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:03.621382 11663 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:03.569988 11890 consensus_queue.cc:237] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.662907 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:03.585665 11730 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.649263 11860 consensus_queue.cc:1048] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260430 07:59:03.658993 11776 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:03.669309 11663 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.667147 11895 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.688438 11776 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:03.702144 11895 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:03.700639 11858 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:03.747859 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 163867us
W20260430 07:59:03.761240 9859 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.007s sys 0.010s
W20260430 07:59:03.762195 9859 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s user 0.008s sys 0.010s
I20260430 07:59:03.628983 9937 rpcz_store.cc:276] 0430 07:59:00.614989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.368710 (+2753721us) service_pool.cc:225] Handling call
0430 07:59:03.381216 (+ 12506us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:03.625385 (+244169us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:03.625658 (+ 273us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:03.765367 10004 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1561) took 3012 ms (client timeout 2604 ms). Trace:
I20260430 07:59:03.765937 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:03.766326 10004 rpcz_store.cc:269] 0430 07:59:00.647289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.647511 (+ 222us) service_pool.cc:225] Handling call
0430 07:59:03.659261 (+3011750us) spinlock_profiling.cc:243] Waited 62.9 ms on lock 0x7b54002c4a54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:03.659397 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144699136}
I20260430 07:59:03.646783 9978 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:03.770547 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 130815us
I20260430 07:59:03.773830 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:03.802057 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:03.809876 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:03.810981 9937 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:59:03.787025 11858 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:03.813578 11915 log.cc:927] Time spent T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.098s user 0.000s sys 0.002s
I20260430 07:59:03.823627 11663 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:03.829656 11854 consensus_queue.cc:1059] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:03.705888 11859 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:03.832049 11854 raft_consensus.cc:3055] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:59:03.832602 11854 raft_consensus.cc:740] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:03.833467 11860 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:03.834419 11860 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:03.835217 11860 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:03.865008 11854 consensus_queue.cc:260] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:03.812948 9828 connection.cc:254] RPC call timeout handler was delayed by -1.5464937589999863s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30010000000000003s for a response
W20260430 07:59:03.917321 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:04.064895 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:03.893437 9828 connection.cc:254] RPC call timeout handler was delayed by -1.3142937589999519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3259s for a response
I20260430 07:59:04.114781 10568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54770 (request call id 304) took 3200 ms. Trace:
W20260430 07:59:04.085870 10566 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54754 (request call id 241) took 3287 ms (client timeout 1863 ms). Trace:
I20260430 07:59:03.940025 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:04.150915 11841 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:04.151839 11841 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:03.982662 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1282) took 3341 ms. Trace:
I20260430 07:59:04.134443 11753 consensus_queue.cc:1048] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.022s
I20260430 07:59:04.154709 9932 rpcz_store.cc:276] 0430 07:59:00.641263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.948275 (+3307012us) service_pool.cc:225] Handling call
0430 07:59:03.980029 (+ 31754us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:03.980873 (+ 844us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:03.980914 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:03.981009 (+ 95us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:03.981063 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:03.981148 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:03.982134 (+ 986us) spinlock_profiling.cc:243] Waited 7.75 ms on lock 0x7b54000b40d0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:03.982324 (+ 190us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2260,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":226,"spinlock_wait_cycles":17816832}
W20260430 07:59:04.122798 11853 consensus_meta.cc:287] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.668s user 0.018s sys 0.002s
W20260430 07:59:04.100415 9935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 885) took 3845 ms (client timeout 1746 ms). Trace:
I20260430 07:59:04.148773 10568 rpcz_store.cc:276] 0430 07:59:00.914354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.099660 (+3185306us) service_pool.cc:225] Handling call
0430 07:59:04.100954 (+ 1294us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:04.113479 (+ 12525us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:04.113509 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:04.113577 (+ 68us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:04.113613 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:04.114166 (+ 553us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:04.114435 (+ 269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:04.154995 9854 raft_consensus.cc:1275] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260430 07:59:03.980324 9931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1274) took 3371 ms (client timeout 1651 ms). Trace:
W20260430 07:59:04.101444 10008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1563) took 3331 ms (client timeout 2675 ms). Trace:
I20260430 07:59:04.113036 9933 raft_consensus.cc:1275] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:59:04.166931 11469 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:04.167366 10563 consensus_meta.cc:287] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.558s user 0.013s sys 0.000s
I20260430 07:59:04.168279 10563 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:04.169952 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:04.086992 10567 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54754 (request call id 240) took 3331 ms (client timeout 1996 ms). Trace:
I20260430 07:59:04.126658 9937 raft_consensus.cc:1240] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
W20260430 07:59:04.149336 10566 rpcz_store.cc:269] 0430 07:59:00.798348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.085436 (+3287088us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:04.085604 (+ 168us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260430 07:59:03.982007 10010 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 929) took 4974 ms (client timeout 1896 ms). Trace:
I20260430 07:59:03.927296 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1327) took 2865 ms. Trace:
I20260430 07:59:04.148070 11854 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:04.177646 11730 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:04.155355 11841 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:03.937533 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:04.180711 11561 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 5.028s user 0.025s sys 0.012s
W20260430 07:59:04.083284 11767 consensus_meta.cc:287] T 805814cfaf2b43ab87c90b2b7b56a211 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.573s user 0.010s sys 0.011s
I20260430 07:59:04.103375 11860 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:03.977391 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:04.086795 11810 consensus_meta.cc:287] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.626s user 0.007s sys 0.013s
W20260430 07:59:04.084698 10562 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54754 (request call id 247) took 3160 ms (client timeout 2067 ms). Trace:
W20260430 07:59:03.975917 9936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 895) took 3675 ms (client timeout 2609 ms). Trace:
W20260430 07:59:04.161595 9935 rpcz_store.cc:269] 0430 07:59:00.256580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.256770 (+ 190us) service_pool.cc:225] Handling call
0430 07:59:03.968004 (+3711234us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b54001ee954. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:04.099679 (+131675us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2255155968}
I20260430 07:59:04.086405 9977 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:03.878110 11876 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:04.163048 9931 rpcz_store.cc:269] 0430 07:59:00.610035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:01.119643 (+509608us) service_pool.cc:225] Handling call
0430 07:59:03.979795 (+2860152us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54000b40d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:03.980036 (+ 241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1146764160}
W20260430 07:59:04.171887 10567 rpcz_store.cc:269] 0430 07:59:00.755560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.086618 (+3331058us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:04.086795 (+ 177us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260430 07:59:04.163269 10008 rpcz_store.cc:269] 0430 07:59:00.770336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.770867 (+ 531us) service_pool.cc:225] Handling call
0430 07:59:03.978987 (+3208120us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b540037fad4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:04.101173 (+122186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3424882560}
I20260430 07:59:04.207374 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1281) took 3550 ms. Trace:
I20260430 07:59:04.267103 9937 rpcz_store.cc:276] 0430 07:59:00.640927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.811890 (+3170963us) service_pool.cc:225] Handling call
0430 07:59:04.126200 (+314310us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:04.190121 (+ 63921us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:04.190823 (+ 702us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:04.180727 10006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1592) took 1149 ms. Trace:
I20260430 07:59:04.174418 9858 rpcz_store.cc:276] 0430 07:59:01.061841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:01.064781 (+ 2940us) service_pool.cc:225] Handling call
0430 07:59:01.065046 (+ 265us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:01.542918 (+477872us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:03.926646 (+2383728us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b5400270bd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:03.927069 (+ 423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1096439680}
I20260430 07:59:04.204548 9929 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:03.878048 11443 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:04.172806 10010 rpcz_store.cc:269] 0430 07:58:59.007700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:59.076174 (+ 68474us) service_pool.cc:225] Handling call
0430 07:59:03.981396 (+4905222us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x7b5400048dd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:03.981654 (+ 258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5199369088}
I20260430 07:59:04.281616 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.284957 11619 consensus_queue.cc:1059] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:04.285246 11571 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.016s
I20260430 07:59:04.285751 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:04.287197 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
W20260430 07:59:04.280619 9929 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1258) took 3712 ms (client timeout 1558 ms). Trace:
I20260430 07:59:04.295862 9828 leader_election.cc:400] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4.
I20260430 07:59:04.230068 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.300915 11876 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:04.230662 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.208395 9930 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:04.297943 11599 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:04.260459 9936 rpcz_store.cc:269] 0430 07:59:00.300070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.347569 (+ 47499us) service_pool.cc:225] Handling call
0430 07:59:01.857524 (+1509955us) spinlock_profiling.cc:243] Waited 6.6 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:03.975189 (+2117665us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54003241d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:03.975603 (+ 414us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462777472}
I20260430 07:59:04.305892 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.313391 11884 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:04.313105 11512 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:04.289402 9929 rpcz_store.cc:269] 0430 07:59:00.568067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.569672 (+ 1605us) service_pool.cc:225] Handling call
0430 07:59:03.937509 (+3367837us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:04.280505 (+342996us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8585984}
I20260430 07:59:04.302625 9828 leader_election.cc:403] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260430 07:59:04.210358 11859 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:04.290078 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 44737.4us
I20260430 07:59:04.274336 10006 rpcz_store.cc:276] 0430 07:59:03.031038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.031797 (+ 759us) service_pool.cc:225] Handling call
0430 07:59:03.032270 (+ 473us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:03.245809 (+213539us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:03.245837 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:03.268232 (+ 22395us) log.cc:844] Serialized 26 byte log entry
0430 07:59:03.554695 (+286463us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:03.554745 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:03.554829 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:04.139294 (+584465us) raft_consensus.cc:1685] finished
0430 07:59:04.148085 (+ 8791us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:04.180237 (+ 32152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3960,"raft.queue_time_us":190281,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":27600,"thread_start_us":424186,"threads_started":2,"wal-append.queue_time_us":294317}
W20260430 07:59:04.261638 10562 rpcz_store.cc:269] 0430 07:59:00.924298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.084123 (+3159825us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:04.084401 (+ 278us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260430 07:59:04.336229 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.336809 9938 raft_consensus.cc:2393] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:04.336478 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:04.378305 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:04.344939 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:04.411466 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:04.308822 11561 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 5.157s user 0.027s sys 0.026s
I20260430 07:59:04.434715 9932 raft_consensus.cc:2393] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:59:04.321151 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.449990 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.459064 9937 raft_consensus.cc:2376] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4. Re-sending same reply.
I20260430 07:59:04.461308 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:04.455793 9938 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 898) took 4110 ms (client timeout 2230 ms). Trace:
I20260430 07:59:04.392406 11323 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:59:04.469233 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:04.467690 9938 rpcz_store.cc:269] 0430 07:59:00.345550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.346202 (+ 652us) service_pool.cc:225] Handling call
0430 07:59:03.976829 (+3630627us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:04.455481 (+478652us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6505344}
I20260430 07:59:04.470953 9931 raft_consensus.cc:2393] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:04.471131 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:04.471920 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.472855 9931 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:04.302413 11810 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:04.407891 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1280) took 3724 ms. Trace:
I20260430 07:59:04.314946 11774 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:04.397073 11769 raft_consensus.cc:3055] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:04.374364 11619 raft_consensus.cc:3055] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:04.483984 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:04.494295 9929 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:04.489998 9936 raft_consensus.cc:2393] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:04.492300 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 406050us
I20260430 07:59:04.496203 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:04.327270 11841 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:04.500722 10564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54754 (request call id 244) took 3632 ms (client timeout 2066 ms). Trace:
I20260430 07:59:04.466403 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:04.505581 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:04.514384 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1268) took 3915 ms. Trace:
W20260430 07:59:04.507198 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 211679us
I20260430 07:59:04.519126 9933 rpcz_store.cc:276] 0430 07:59:00.598593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.273413 (+2674820us) service_pool.cc:225] Handling call
0430 07:59:04.111729 (+838316us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:04.173045 (+ 61316us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:04.514129 (+341084us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x7b54001ee950. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:04.514292 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1903253760}
I20260430 07:59:04.314651 11884 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:04.492969 11619 raft_consensus.cc:740] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:59:04.512321 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 107835us
I20260430 07:59:04.520120 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:04.401870 11512 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:04.490929 11769 raft_consensus.cc:740] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:04.489589 9930 rpcz_store.cc:276] 0430 07:59:00.640636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.810204 (+3169568us) service_pool.cc:225] Handling call
0430 07:59:04.207524 (+397320us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:04.315109 (+107585us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:04.365073 (+ 49964us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:04.526680 9929 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:04.533077 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:04.501255 10564 rpcz_store.cc:269] 0430 07:59:00.868011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.500361 (+3632350us) service_pool.cc:212] Skipping call since client already timed out
0430 07:59:04.500529 (+ 168us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260430 07:59:04.537225 9937 raft_consensus.cc:2410] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260430 07:59:04.501910 11774 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:04.521001 11860 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:04.545378 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 74590.7us
I20260430 07:59:04.517855 9932 raft_consensus.cc:2376] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4. Re-sending same reply.
I20260430 07:59:04.543972 9935 raft_consensus.cc:2376] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5. Re-sending same reply.
I20260430 07:59:04.565802 11925 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4.
I20260430 07:59:04.567451 11512 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:04.556763 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.527132 11841 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:04.576643 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.581686 11860 consensus_queue.cc:1059] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:04.583043 11860 raft_consensus.cc:3055] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:04.582299 9857 raft_consensus.cc:1240] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:04.587219 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:04.587467 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:04.591780 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.588707 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.590036 9938 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:04.596699 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 67879us
I20260430 07:59:04.601892 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.583473 11860 raft_consensus.cc:740] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:04.610160 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.614221 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.613890 11769 consensus_queue.cc:260] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:04.615165 9858 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:04.616122 9903 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:04.617228 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:04.618218 11599 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:04.618708 11599 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:04.619216 11599 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:04.610211 9935 raft_consensus.cc:2376] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5. Re-sending same reply.
I20260430 07:59:04.616753 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.624830 9858 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:04.627734 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.629207 9858 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:04.645671 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.645864 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:04.646479 9863 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:04.646800 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:04.648000 9858 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 6
I20260430 07:59:04.637938 9857 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:04.650240 11884 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:04.640272 11841 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:04.651213 11810 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:04.653401 11774 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:04.603356 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1346) took 1249 ms. Trace:
I20260430 07:59:04.654312 9856 rpcz_store.cc:276] 0430 07:59:03.245255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.246346 (+ 1091us) service_pool.cc:225] Handling call
0430 07:59:03.279902 (+ 33556us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:03.280214 (+ 312us) raft_consensus.cc:1534] Early marking committed up to index 4
0430 07:59:03.749501 (+469287us) log.cc:844] Serialized 14 byte log entry
0430 07:59:04.315995 (+566494us) log.cc:844] Serialized 14 byte log entry
0430 07:59:04.445252 (+129257us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:04.449565 (+ 4313us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:04.467015 (+ 17450us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:04.491463 (+ 24448us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:04.494566 (+ 3103us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":98673,"prepare.queue_time_us":807569,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":568595,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":11080,"thread_start_us":705855,"threads_started":4,"wal-append.queue_time_us":548160}
I20260430 07:59:04.633744 11876 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:04.644555 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:04.647809 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.641453 11619 consensus_queue.cc:260] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:04.625108 9977 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:04.666559 9936 raft_consensus.cc:2410] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260430 07:59:04.670982 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 46046.2us
I20260430 07:59:04.671818 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:04.670326 9857 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:04.675909 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:04.685714 10033 debug-util.cc:398] Leaking SignalData structure 0x7b08003c9880 after lost signal to thread 10036
I20260430 07:59:04.687075 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1397) took 8160 ms. Trace:
I20260430 07:59:04.687866 9855 rpcz_store.cc:276] 0430 07:58:56.507726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:59.268947 (+2761221us) service_pool.cc:225] Handling call
0430 07:58:59.661435 (+392488us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:58:59.771819 (+110384us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:04.667911 (+4896092us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b5400208d50. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:04.668209 (+ 298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":874265984}
I20260430 07:59:04.693815 11810 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:04.694788 11810 raft_consensus.cc:697] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:04.698068 9938 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 2.
I20260430 07:59:04.700085 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.703514 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.706627 9977 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:04.685895 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.717767 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.696389 11810 consensus_queue.cc:237] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:04.712337 9934 raft_consensus.cc:2376] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5. Re-sending same reply.
I20260430 07:59:04.613075 11860 consensus_queue.cc:260] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:04.726966 9934 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1283) took 4085 ms (client timeout 1685 ms). Trace:
W20260430 07:59:04.728750 9934 rpcz_store.cc:269] 0430 07:59:00.641590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:02.209643 (+1568053us) service_pool.cc:225] Handling call
0430 07:59:04.679458 (+2469815us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:04.726890 (+ 47432us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23110528}
W20260430 07:59:04.745827 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:04.713528 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.754212 9931 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:04.750675 9978 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:04.750402 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.758374 11744 consensus_queue.cc:1059] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260430 07:59:04.759946 11780 raft_consensus.cc:3055] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:04.760535 11780 raft_consensus.cc:740] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:04.761914 11780 consensus_queue.cc:260] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:04.763700 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:04.770112 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 44866.7us
I20260430 07:59:04.776764 11892 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:04.777597 11892 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:04.711998 11841 consensus_queue.cc:1059] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260430 07:59:04.779613 9938 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:04.773803 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:04.785035 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:04.786935 9978 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:04.788681 9863 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 5
I20260430 07:59:04.799878 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.786779 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:04.802879 10533 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 5.638s user 0.004s sys 0.022s
I20260430 07:59:04.803414 10561 raft_consensus.cc:3055] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:59:04.803710 11841 raft_consensus.cc:3055] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:04.803881 10561 raft_consensus.cc:740] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:04.804298 11841 raft_consensus.cc:740] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:04.804582 11892 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:04.797501 11655 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:04.806015 11655 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:04.808501 11655 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:04.810058 10010 raft_consensus.cc:1240] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:04.812170 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:04.812630 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 140839us
I20260430 07:59:04.814028 9977 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:04.817200 11892 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:04.821342 11890 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:04.822199 11890 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:04.824229 10003 raft_consensus.cc:1240] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:04.824838 11890 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:04.821651 11647 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260430 07:59:04.825495 11859 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:04.825810 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:04.826716 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.828049 11854 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:04.830005 9932 raft_consensus.cc:2368] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:04.831808 9863 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:04.832273 9977 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:04.793025 11780 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:04.834928 9978 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:04.845609 10533 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 5.684s user 0.005s sys 0.022s
I20260430 07:59:04.845742 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.845621 10010 raft_consensus.cc:1275] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:59:04.836472 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.863090 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:04.863500 9858 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:04.869331 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.841495 11859 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:04.805171 10561 consensus_queue.cc:260] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:04.877421 9859 raft_consensus.cc:2393] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:04.843320 11854 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:04.863875 9936 raft_consensus.cc:2376] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4. Re-sending same reply.
W20260430 07:59:04.878005 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 45837.1us
W20260430 07:59:04.882300 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 87810.8us
I20260430 07:59:04.885998 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:04.893643 9938 raft_consensus.cc:2376] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4. Re-sending same reply.
I20260430 07:59:04.895210 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.896061 9938 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
W20260430 07:59:04.896766 9977 leader_election.cc:336] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.536s (SENT)
I20260430 07:59:04.897518 9977 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:04.904242 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.901201 9859 raft_consensus.cc:1275] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:59:04.905133 9863 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:59:04.902750 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:04.908442 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.879995 11655 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:04.903448 11860 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:04.911608 11720 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:04.912246 11720 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:04.912340 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:04.912667 11720 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:04.914461 9932 raft_consensus.cc:1275] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:04.914845 10009 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:04.915269 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1285) took 4272 ms. Trace:
I20260430 07:59:04.915709 9932 rpcz_store.cc:276] 0430 07:59:00.642510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.913695 (+4271185us) service_pool.cc:225] Handling call
0430 07:59:04.914064 (+ 369us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:04.914969 (+ 905us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:04.915198 (+ 229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:04.916751 9859 raft_consensus.cc:1275] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:04.919601 10004 raft_consensus.cc:1275] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:59:04.880049 11859 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:04.930714 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:04.902405 11854 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:04.910946 9903 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.132:40723 timed out after 1.568s (SENT) [suppressed 54 similar messages]
I20260430 07:59:04.937811 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:04.938752 9863 raft_consensus.cc:2368] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5.
I20260430 07:59:04.920593 9932 raft_consensus.cc:1240] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260430 07:59:04.940156 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 68158.2us
I20260430 07:59:04.940737 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:04.908402 11909 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:04.950013 9856 raft_consensus.cc:2368] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6.
W20260430 07:59:04.948344 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 63409.2us
I20260430 07:59:04.957584 9862 raft_consensus.cc:1275] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:04.936610 10561 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:04.960038 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.966033 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.877228 11841 consensus_queue.cc:260] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:04.921197 11599 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:04.977217 9859 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:04.981940 11841 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:05.034485 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:05.035817 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:04.990700 9978 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:04.992755 11725 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:05.037904 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 47312.1us
I20260430 07:59:05.038553 11725 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:05.038873 9978 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:05.039270 11725 raft_consensus.cc:697] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:05.039840 9978 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:04.993961 11730 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:05.041147 11730 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:05.043681 11730 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:04.994473 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1319) took 4624 ms. Trace:
I20260430 07:59:04.995170 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:04.996081 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.045941 10559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54754 (request call id 251) took 1457 ms. Trace:
I20260430 07:59:04.983326 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:04.999158 11817 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:05.002449 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.002898 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:05.003801 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:05.005375 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 108645us
I20260430 07:59:05.007047 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.008646 9858 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:04.996140 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:05.017392 11599 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:05.024210 11647 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:05.024993 11654 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:05.035352 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:05.047032 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:04.998157 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.047611 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:05.046334 10559 rpcz_store.cc:276] 0430 07:59:03.588670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.598148 (+ 9478us) service_pool.cc:225] Handling call
0430 07:59:04.994884 (+1396736us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b58001f1798. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:05.045855 (+ 50971us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4017920}
W20260430 07:59:05.063126 10533 connection.cc:254] RPC call timeout handler was delayed by -3.9700519849999978s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
W20260430 07:59:05.077378 10533 connection.cc:254] RPC call timeout handler was delayed by -3.9700519849999978s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
I20260430 07:59:05.079686 9862 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:05.040593 11725 consensus_queue.cc:237] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.083897 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:05.089659 9977 leader_election.cc:336] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.601s (SENT)
I20260430 07:59:05.088407 11854 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.095044 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:05.098362 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:05.099920 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.044487 9861 rpcz_store.cc:276] 0430 07:59:00.369783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:00.567302 (+197519us) service_pool.cc:225] Handling call
0430 07:59:00.587761 (+ 20459us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:00.588100 (+ 339us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:59:00.602031 (+ 13931us) log.cc:844] Serialized 14 byte log entry
0430 07:59:04.688661 (+4086630us) spinlock_profiling.cc:243] Waited 6.76 ms on lock 0x7b18003b3860. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661c8587 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e5242 00007fb568e1c5ab 00007fb568e220c3 00007fb568e225da 00007fb56928de0a
0430 07:59:04.837066 (+148405us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:04.882506 (+ 45440us) log.cc:844] Serialized 26 byte log entry
0430 07:59:04.885878 (+ 3372us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:04.885928 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:04.886036 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:04.970080 (+ 84044us) raft_consensus.cc:1685] finished
0430 07:59:04.973623 (+ 3543us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:04.994267 (+ 20644us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13997,"prepare.queue_time_us":16894,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":1862,"raft.queue_time_us":41540,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":6539,"spinlock_wait_cycles":15538944,"thread_start_us":79705,"threads_started":1,"wal-append.queue_time_us":4287574}
I20260430 07:59:05.107856 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.097043 11854 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:05.108350 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1287) took 4458 ms. Trace:
I20260430 07:59:05.117108 9859 raft_consensus.cc:2410] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:59:05.117550 9863 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:05.088819 11655 consensus_queue.cc:1048] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
I20260430 07:59:05.072475 11810 consensus_queue.cc:1048] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260430 07:59:05.118206 10008 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:05.126135 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:05.032847 11720 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.103456 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:05.130625 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.132366 9931 raft_consensus.cc:2368] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4.
I20260430 07:59:05.129637 10533 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:05.114876 9936 rpcz_store.cc:276] 0430 07:59:00.643060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.893453 (+4250393us) service_pool.cc:225] Handling call
0430 07:59:04.893923 (+ 470us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:04.894427 (+ 504us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:59:04.894823 (+ 396us) log.cc:844] Serialized 14 byte log entry
0430 07:59:05.100841 (+206018us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:05.100929 (+ 88us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:05.101003 (+ 74us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:05.101103 (+ 100us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:05.101509 (+ 406us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11658,"prepare.queue_time_us":185,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":13202,"raft.run_cpu_time_us":398,"raft.run_wall_time_us":42937,"thread_start_us":205340,"threads_started":1}
I20260430 07:59:05.119596 9858 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 4: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:05.127254 9859 raft_consensus.cc:1240] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:05.090930 11647 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.079339 11817 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.143213 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:05.158416 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 54218.4us
I20260430 07:59:05.232264 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1308) took 1115 ms. Trace:
I20260430 07:59:05.155227 10003 raft_consensus.cc:1240] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:05.238968 9938 rpcz_store.cc:276] 0430 07:59:04.117130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.897031 (+779901us) service_pool.cc:225] Handling call
0430 07:59:04.903148 (+ 6117us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:04.968350 (+ 65202us) raft_consensus.cc:1534] Early marking committed up to index 3
0430 07:59:04.979545 (+ 11195us) log.cc:844] Serialized 14 byte log entry
0430 07:59:05.231166 (+251621us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:05.231229 (+ 63us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:05.231286 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:05.231359 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:05.232172 (+ 813us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7330,"prepare.queue_time_us":17765,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":108651,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":105,"thread_start_us":142569,"threads_started":1}
I20260430 07:59:05.247875 11850 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:59:05.250974 9828 leader_election.cc:336] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.540s (SENT)
W20260430 07:59:05.152897 11930 log.cc:927] Time spent T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.350s user 0.003s sys 0.000s
I20260430 07:59:05.248545 11850 raft_consensus.cc:697] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:05.153033 11720 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:05.233017 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:05.265029 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:05.266667 11752 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.154s
W20260430 07:59:05.274663 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 363597us
W20260430 07:59:05.268958 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:05.263103 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.272284 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.274874 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.153422 11565 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:05.279320 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:05.280119 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 208454us
I20260430 07:59:05.286687 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.295569 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:05.296892 9859 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:05.300375 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:05.138695 11860 consensus_queue.cc:1048] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.149s
I20260430 07:59:05.294631 9932 raft_consensus.cc:2410] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:59:05.303252 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:05.305373 9856 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:05.288544 10033 debug-util.cc:398] Leaking SignalData structure 0x7b08003ca120 after lost signal to thread 10595
I20260430 07:59:05.305547 9931 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:05.138556 11654 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.294361 11712 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:05.308292 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.312005 11712 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:05.312489 11712 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:05.309897 9938 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:05.316393 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:05.309499 10033 debug-util.cc:398] Leaking SignalData structure 0x7b08003ca040 after lost signal to thread 10597
I20260430 07:59:05.303823 11565 raft_consensus.cc:697] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:05.323683 9858 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:05.323129 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:05.329900 9931 raft_consensus.cc:2393] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:05.328850 10533 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:05.342111 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:05.301893 11859 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:05.349040 9903 connection.cc:254] RPC call timeout handler was delayed by -1.2588656849999893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.217s for a response
W20260430 07:59:05.349823 9903 connection.cc:254] RPC call timeout handler was delayed by -1.1914461650000021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2114s for a response
I20260430 07:59:05.356079 9936 raft_consensus.cc:2393] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:05.366165 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.359330 9978 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:05.368357 11654 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:05.369383 11647 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:05.371747 11867 log.cc:927] Time spent T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 3.391s user 0.007s sys 0.007s
I20260430 07:59:05.351588 11831 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:05.391263 9938 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:05.395274 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:05.397006 9934 raft_consensus.cc:2376] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4. Re-sending same reply.
I20260430 07:59:05.411458 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1305) took 1288 ms. Trace:
I20260430 07:59:05.412045 9934 rpcz_store.cc:276] 0430 07:59:04.114689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.742728 (+628039us) service_pool.cc:225] Handling call
0430 07:59:05.395107 (+652379us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8a3e 000000000035a3c9 00007fb56e1e08e0 00007fb56e2a9784 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:05.402605 (+ 7498us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6763264}
W20260430 07:59:05.390933 11780 consensus_meta.cc:287] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.516s user 0.000s sys 0.014s
I20260430 07:59:05.418823 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.400822 11831 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:05.429497 11827 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:05.429836 10033 debug-util.cc:398] Leaking SignalData structure 0x7b08003cb640 after lost signal to thread 11631
I20260430 07:59:05.386291 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:05.363260 11850 consensus_queue.cc:237] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:05.444011 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 48342.2us
W20260430 07:59:05.447849 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 180650us
I20260430 07:59:05.450565 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:05.455601 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.447606 11831 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:05.451072 11827 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:05.435778 11565 consensus_queue.cc:237] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.457600 9934 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:05.470255 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.472170 9936 raft_consensus.cc:2393] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:05.459082 11817 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:05.478461 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:05.474340 11946 log.cc:927] Time spent T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260430 07:59:05.470721 11827 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:05.486023 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.482211 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.487279 9934 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:05.489782 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.496886 9936 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:05.486042 11817 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:05.501432 9828 connection.cc:254] RPC call timeout handler was delayed by -1.4134921439999744s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2449s for a response
I20260430 07:59:05.501729 9862 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
W20260430 07:59:05.502651 9828 connection.cc:254] RPC call timeout handler was delayed by -1.281192143999931s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2596s for a response
I20260430 07:59:05.509418 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:05.512147 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 212773us
I20260430 07:59:05.511613 9862 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:05.500912 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.498198 11859 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:05.525837 9977 leader_election.cc:336] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.638s (SENT)
I20260430 07:59:05.526537 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:05.540133 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 48961.2us
I20260430 07:59:05.564227 9938 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:05.566606 9936 raft_consensus.cc:2393] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:05.567690 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.504145 11817 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.569417 9932 raft_consensus.cc:2393] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:05.578950 9859 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:05.584537 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 61411.4us
I20260430 07:59:05.585628 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.587483 9978 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:05.591274 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.593127 9903 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:05.594949 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.595590 9938 raft_consensus.cc:2393] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:05.597391 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.602579 9862 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:05.605161 9978 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:05.606988 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.607548 9938 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
W20260430 07:59:05.608601 11943 log.cc:927] Time spent T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.094s user 0.002s sys 0.000s
W20260430 07:59:05.614049 9860 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1345) took 2359 ms (client timeout 1961 ms). Trace:
I20260430 07:59:05.614494 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1572) took 4243 ms. Trace:
W20260430 07:59:05.614853 9860 rpcz_store.cc:269] 0430 07:59:03.244929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:03.245070 (+ 141us) service_pool.cc:225] Handling call
0430 07:59:05.578904 (+2333834us) spinlock_profiling.cc:243] Waited 27.7 ms on lock 0x7b540010c054. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:05.604685 (+ 25781us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":63752064}
I20260430 07:59:05.615165 10005 rpcz_store.cc:276] 0430 07:59:01.374694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:01.374867 (+ 173us) service_pool.cc:225] Handling call
0430 07:59:01.410218 (+ 35351us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:03.323739 (+1913521us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:05.587160 (+2263421us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b54000492d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:05.605727 (+ 18567us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4251923200}
W20260430 07:59:05.617607 9977 leader_election.cc:336] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.695s (SENT)
W20260430 07:59:05.619488 9977 leader_election.cc:336] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.725s (SENT)
I20260430 07:59:05.598469 11722 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:59:05.621156 9977 leader_election.cc:336] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.803s (SENT)
I20260430 07:59:05.633736 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:05.634689 9856 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
W20260430 07:59:05.634147 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.625826 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:05.617643 11826 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:59:05.640158 9929 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1331) took 1466 ms (client timeout 1803 ms). Trace:
W20260430 07:59:05.640812 9929 rpcz_store.cc:269] 0430 07:59:04.167966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.555423 (+387457us) service_pool.cc:225] Handling call
0430 07:59:05.631664 (+1076241us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:05.634434 (+ 2770us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4958976}
I20260430 07:59:05.636572 9978 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:05.642167 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.642835 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.654101 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.621523 11742 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:05.629384 11865 log.cc:927] Time spent T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 3.608s user 0.013s sys 0.004s
I20260430 07:59:05.627449 11722 raft_consensus.cc:697] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:05.658380 11859 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election won for term 6
W20260430 07:59:05.660789 11945 log.cc:927] Time spent T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.169s user 0.001s sys 0.001s
I20260430 07:59:05.661137 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1573) took 4272 ms. Trace:
I20260430 07:59:05.661890 9860 raft_consensus.cc:2393] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:05.658283 11722 consensus_queue.cc:237] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.660164 11817 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:05.661811 10007 rpcz_store.cc:276] 0430 07:59:01.389107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:01.389259 (+ 152us) service_pool.cc:225] Handling call
0430 07:59:01.389699 (+ 440us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:01.395990 (+ 6291us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:01.396041 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:01.402560 (+ 6519us) log.cc:844] Serialized 26 byte log entry
0430 07:59:02.013891 (+611331us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:02.014165 (+ 274us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:02.014287 (+ 122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:05.479277 (+3464990us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:05.660490 (+181213us) raft_consensus.cc:1685] finished
0430 07:59:05.660515 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:05.660893 (+ 378us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5766,"raft.queue_time_us":147596,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":4294,"spinlock_wait_cycles":23325568,"thread_start_us":610882,"threads_started":1,"wal-append.queue_time_us":562515}
W20260430 07:59:05.668948 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 93921.1us
I20260430 07:59:05.676977 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.678279 9938 raft_consensus.cc:2410] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:59:05.668402 11859 raft_consensus.cc:697] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:05.679329 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.682900 9936 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:05.685896 9929 raft_consensus.cc:2393] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:05.687631 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.688436 9936 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:05.666003 11826 raft_consensus.cc:697] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:05.693639 9938 raft_consensus.cc:1240] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 2. Current term is 3. Ops: []
W20260430 07:59:05.704133 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.719628 9929 raft_consensus.cc:1240] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:05.720631 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1313) took 1592 ms. Trace:
I20260430 07:59:05.721035 9929 rpcz_store.cc:276] 0430 07:59:04.128115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:05.686947 (+1558832us) service_pool.cc:225] Handling call
0430 07:59:05.695037 (+ 8090us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:05.720259 (+ 25222us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:05.720545 (+ 286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:05.722289 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1312) took 1567 ms. Trace:
I20260430 07:59:05.722985 9938 rpcz_store.cc:276] 0430 07:59:04.127665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:05.680980 (+1553315us) service_pool.cc:225] Handling call
0430 07:59:05.693407 (+ 12427us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:05.694274 (+ 867us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:05.694532 (+ 258us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:05.727116 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 105958us
I20260430 07:59:05.734494 11712 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:05.723663 11857 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:05.738402 11857 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.739673 11712 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:05.687390 11859 consensus_queue.cc:237] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.744483 9903 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:05.748124 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:05.750308 9977 leader_election.cc:336] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.862s (SENT)
I20260430 07:59:05.751230 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.751933 9938 raft_consensus.cc:2393] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:05.760396 9963 heartbeater.cc:646] Failed to heartbeat to 127.8.119.190:40689 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.8.119.190:40689 timed out after 15.000s (SENT)
W20260430 07:59:05.764508 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.768102 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:05.741226 11857 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:05.773988 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:05.778335 9938 raft_consensus.cc:1275] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:05.779556 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1316) took 1650 ms. Trace:
I20260430 07:59:05.780259 9938 rpcz_store.cc:276] 0430 07:59:04.129539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:05.753074 (+1623535us) service_pool.cc:225] Handling call
0430 07:59:05.772524 (+ 19450us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:05.779177 (+ 6653us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:05.779471 (+ 294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:05.782193 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:05.729988 11827 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.789937 9860 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:59:05.776789 11769 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:05.791399 9856 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
W20260430 07:59:05.794100 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 43812.8us
W20260430 07:59:05.794999 9977 leader_election.cc:336] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.891s (SENT)
I20260430 07:59:05.799360 11826 consensus_queue.cc:237] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.801731 9978 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:05.777331 11817 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:05.806093 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:05.808027 9977 leader_election.cc:336] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.226s (SENT)
W20260430 07:59:05.809232 9977 leader_election.cc:336] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.234s (SENT)
I20260430 07:59:05.826289 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:05.827088 9938 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:05.830380 9860 raft_consensus.cc:1275] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:59:05.815677 11722 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:05.813163 11831 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:05.846029 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 101629us
I20260430 07:59:05.846738 11827 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:05.847429 9858 raft_consensus.cc:1275] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:05.848589 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:05.850184 9856 raft_consensus.cc:1240] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:05.850045 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:05.855751 9858 raft_consensus.cc:2410] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:59:05.861104 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:05.863231 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:05.863899 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.867336 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:05.868271 9933 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1332) took 1700 ms (client timeout 1863 ms). Trace:
W20260430 07:59:05.868664 9933 rpcz_store.cc:269] 0430 07:59:04.168396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.595172 (+426776us) service_pool.cc:225] Handling call
0430 07:59:05.867202 (+1272030us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8a3e 000000000035a3c9 00007fb56e1e08e0 00007fb56e2a9784 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:05.868201 (+ 999us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48941440}
I20260430 07:59:05.869925 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:05.870136 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.870630 9930 raft_consensus.cc:2376] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5. Re-sending same reply.
W20260430 07:59:05.871167 9930 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1300) took 1878 ms (client timeout 1862 ms). Trace:
I20260430 07:59:05.871281 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:05.871567 9930 rpcz_store.cc:269] 0430 07:59:03.993139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.545837 (+552698us) service_pool.cc:225] Handling call
0430 07:59:05.869755 (+1323918us) spinlock_profiling.cc:243] Waited 6.79 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:05.871105 (+ 1350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15615360}
I20260430 07:59:05.873236 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.875219 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.876137 9930 raft_consensus.cc:2393] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:05.877254 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.882509 11831 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:05.872097 9933 raft_consensus.cc:2393] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:05.891064 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.895612 11619 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:05.841779 11722 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:05.898854 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.900626 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:05.910586 9934 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:05.901659 9858 raft_consensus.cc:2410] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:59:05.902030 11807 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:05.903151 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:05.903446 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:05.904412 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:05.843675 11859 consensus_queue.cc:1048] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.063s
I20260430 07:59:05.917003 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:05.916822 11807 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.917613 11859 consensus_queue.cc:1243] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer 77e19ab653f540339c1c613fad1451b5 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:59:05.917779 10012 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:05.918134 11804 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:05.918618 11804 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:05.919144 11807 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:05.920177 11964 log.cc:927] Time spent T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
W20260430 07:59:05.898011 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.910574 9930 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:59:05.921139 11804 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:05.916074 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:05.916023 11520 consensus_queue.cc:1048] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260430 07:59:05.922487 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.916335 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.920269 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:05.906005 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 47183.2us
I20260430 07:59:05.910838 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:05.930310 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:05.925745 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.926723 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.928283 9857 raft_consensus.cc:1240] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:05.923738 9934 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:05.926121 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:05.924141 9858 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:05.934512 10011 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
W20260430 07:59:05.942430 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.962342 9978 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:05.923669 9930 raft_consensus.cc:2410] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260430 07:59:05.940599 9935 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:59:05.974714 9935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1320) took 1843 ms (client timeout 1938 ms). Trace:
W20260430 07:59:05.975119 9935 rpcz_store.cc:269] 0430 07:59:04.131618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.623397 (+491779us) service_pool.cc:225] Handling call
0430 07:59:05.925487 (+1302090us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:05.974652 (+ 49165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48425344}
W20260430 07:59:05.970870 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:05.972687 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:05.978015 10007 raft_consensus.cc:1240] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:05.980252 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:05.985251 10007 raft_consensus.cc:2393] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:05.925709 11807 consensus_queue.cc:1059] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260430 07:59:05.987046 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.987358 11807 raft_consensus.cc:3055] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:05.987813 9933 raft_consensus.cc:2376] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4. Re-sending same reply.
I20260430 07:59:05.988775 10010 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:05.979354 11857 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:05.989711 10010 raft_consensus.cc:606] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:05.989988 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:05.990749 10011 raft_consensus.cc:2368] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:05.987855 11807 raft_consensus.cc:740] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:05.989094 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.998773 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.993759 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:05.925192 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:05.997411 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:05.913084 11722 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:06.001928 9937 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:05.927348 11520 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260430 07:59:06.004500 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:06.004956 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:06.005961 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:06.006255 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:06.006755 9930 raft_consensus.cc:2393] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:06.007200 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:05.926409 11891 consensus_queue.cc:1048] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.083s
W20260430 07:59:06.009958 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:06.010500 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:06.011216 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.011904 10011 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
I20260430 07:59:06.012427 11807 consensus_queue.cc:260] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.014202 11807 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:06.014918 9977 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:05.989733 11857 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:06.015753 11857 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:06.017395 9935 raft_consensus.cc:2393] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:06.017534 11859 consensus_queue.cc:1048] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:59:06.018697 9935 raft_consensus.cc:1275] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:06.019080 11859 raft_consensus.cc:2749] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:06.019301 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1319) took 1888 ms. Trace:
I20260430 07:59:06.019761 9935 rpcz_store.cc:276] 0430 07:59:04.131093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.018027 (+1886934us) service_pool.cc:225] Handling call
0430 07:59:06.018377 (+ 350us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:06.019032 (+ 655us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:06.019210 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:06.020426 11797 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:06.021036 11797 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.013022 10006 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:06.023132 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:06.012759 11765 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:06.023761 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1600) took 1180 ms. Trace:
I20260430 07:59:06.034384 10008 rpcz_store.cc:276] 0430 07:59:04.819561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.825482 (+ 5921us) service_pool.cc:225] Handling call
0430 07:59:06.000165 (+1174683us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54002c5454. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:06.000308 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":635422976}
I20260430 07:59:06.021054 9935 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:06.007347 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:06.017019 9977 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:06.036562 9938 raft_consensus.cc:2468] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:06.038048 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 199) took 1090 ms. Trace:
I20260430 07:59:06.038636 9934 rpcz_store.cc:276] 0430 07:59:04.932489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.021731 (+1089242us) service_pool.cc:225] Handling call
0430 07:59:06.022080 (+ 349us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:06.022257 (+ 177us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:06.022281 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:06.022326 (+ 45us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:06.022358 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:06.022424 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:06.022543 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:05.985747 11804 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:06.047241 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 198) took 1153 ms. Trace:
I20260430 07:59:06.050763 9935 rpcz_store.cc:276] 0430 07:59:04.893377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.020546 (+1127169us) service_pool.cc:225] Handling call
0430 07:59:06.020872 (+ 326us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:06.046850 (+ 25978us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:06.047159 (+ 309us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:05.994635 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:06.046053 10533 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:59:06.091969 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 46016.6us
I20260430 07:59:06.055859 9938 raft_consensus.cc:1240] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:06.092701 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1315) took 1963 ms. Trace:
I20260430 07:59:06.093802 11831 raft_consensus.cc:2804] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:06.094527 11831 raft_consensus.cc:697] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:06.094813 11804 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.095718 11831 consensus_queue.cc:237] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.097813 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.093396 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 204) took 1096 ms. Trace:
I20260430 07:59:06.061470 9904 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:06.010025 11501 consensus_queue.cc:1059] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20260430 07:59:06.069872 9933 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:06.081399 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:06.043255 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:06.097708 11780 consensus_queue.cc:1059] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:06.099792 9938 rpcz_store.cc:276] 0430 07:59:04.997047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.053934 (+1056887us) service_pool.cc:225] Handling call
0430 07:59:06.055426 (+ 1492us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:06.093080 (+ 37654us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:06.093317 (+ 237us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:06.107963 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 55139.4us
I20260430 07:59:06.079245 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:06.072409 11797 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:06.107445 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 70944.3us
I20260430 07:59:06.109120 9931 raft_consensus.cc:1240] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:06.117219 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 203) took 1120 ms. Trace:
I20260430 07:59:06.117599 9931 rpcz_store.cc:276] 0430 07:59:04.996713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.051299 (+1054586us) service_pool.cc:225] Handling call
0430 07:59:06.051709 (+ 410us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:06.116986 (+ 65277us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:06.117164 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:06.118634 9933 raft_consensus.cc:1240] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260430 07:59:06.093370 9929 rpcz_store.cc:276] 0430 07:59:04.129051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:05.725390 (+1596339us) service_pool.cc:225] Handling call
0430 07:59:05.748913 (+ 23523us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:05.761934 (+ 13021us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:05.800614 (+ 38680us) log.cc:844] Serialized 14 byte log entry
0430 07:59:05.912145 (+111531us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:05.943065 (+ 30920us) log.cc:844] Serialized 26 byte log entry
0430 07:59:05.943308 (+ 243us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:05.943793 (+ 485us) log.cc:844] Serialized 14 byte log entry
0430 07:59:05.944090 (+ 297us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:05.944198 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:06.092411 (+148213us) raft_consensus.cc:1685] finished
0430 07:59:06.092442 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:06.092621 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3612,"prepare.queue_time_us":144610,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":6380,"raft.queue_time_us":25329,"raft.run_cpu_time_us":740,"raft.run_wall_time_us":29554,"spinlock_wait_cycles":15961984,"thread_start_us":125333,"threads_started":2,"wal-append.queue_time_us":273435}
I20260430 07:59:06.115926 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1314) took 1987 ms. Trace:
I20260430 07:59:06.119902 9932 raft_consensus.cc:1275] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260430 07:59:06.114528 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 57998.8us
W20260430 07:59:06.122696 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 87753.6us
I20260430 07:59:06.120553 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 201) took 1148 ms. Trace:
I20260430 07:59:06.123461 9932 rpcz_store.cc:276] 0430 07:59:04.972359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.038194 (+1065835us) service_pool.cc:225] Handling call
0430 07:59:06.114239 (+ 76045us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:06.120350 (+ 6111us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:06.120511 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:06.125175 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 908) took 1032 ms. Trace:
I20260430 07:59:06.125677 9933 rpcz_store.cc:276] 0430 07:59:05.102529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.101773 (+999244us) service_pool.cc:225] Handling call
0430 07:59:06.118400 (+ 16627us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:06.124919 (+ 6519us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:06.125124 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:06.119657 9936 rpcz_store.cc:276] 0430 07:59:04.128499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:05.692212 (+1563713us) service_pool.cc:225] Handling call
0430 07:59:05.728703 (+ 36491us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:05.769327 (+ 40624us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:05.769358 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:05.786491 (+ 17133us) log.cc:844] Serialized 26 byte log entry
0430 07:59:05.905227 (+118736us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:06.113559 (+208332us) log.cc:844] Serialized 14 byte log entry
0430 07:59:06.115472 (+ 1913us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:06.115570 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:06.115624 (+ 54us) raft_consensus.cc:1685] finished
0430 07:59:06.115635 (+ 11us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:06.115871 (+ 236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5987,"prepare.queue_time_us":19766,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":7825,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":11595,"thread_start_us":279027,"threads_started":2,"wal-append.queue_time_us":285306}
I20260430 07:59:06.127151 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.128962 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:06.130539 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.115808 11619 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:59:06.134135 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 117165us
I20260430 07:59:06.144481 9934 raft_consensus.cc:1240] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:59:06.141727 9932 raft_consensus.cc:1240] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:06.146485 10010 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:06.148308 9936 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:06.148977 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 202) took 1152 ms. Trace:
I20260430 07:59:06.149379 9934 rpcz_store.cc:276] 0430 07:59:04.996311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.053629 (+1057318us) service_pool.cc:225] Handling call
0430 07:59:06.122851 (+ 69222us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:06.148745 (+ 25894us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:06.148923 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:06.151939 11879 consensus_queue.cc:1059] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260430 07:59:06.153662 11724 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:06.154083 11724 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:06.154448 11724 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:06.156016 11722 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.159992 9938 raft_consensus.cc:1275] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:59:06.164721 9929 raft_consensus.cc:1275] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:06.171048 10003 raft_consensus.cc:1275] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:59:06.138516 9935 raft_consensus.cc:2955] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:06.153087 11879 raft_consensus.cc:3055] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:06.145344 11804 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:06.177323 9904 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:06.181185 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.180305 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:06.139586 11780 consensus_queue.cc:1059] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20260430 07:59:06.190469 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 54890.2us
I20260430 07:59:06.191171 9904 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:06.144702 11750 consensus_queue.cc:1048] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.038s
I20260430 07:59:06.197983 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.177785 11879 raft_consensus.cc:740] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:06.212343 10010 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:06.177506 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:06.197297 11789 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:06.169869 11738 consensus_queue.cc:1059] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.031s
I20260430 07:59:06.176653 11742 consensus_queue.cc:1048] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260430 07:59:06.176422 11849 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260430 07:59:06.226166 11859 raft_consensus.cc:3055] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:06.226702 11859 raft_consensus.cc:740] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:06.227655 11724 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.193569 11601 consensus_queue.cc:1059] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20260430 07:59:06.209076 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.214529 9904 leader_election.cc:400] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Vote denied by peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679) with higher term. Message: Invalid argument: T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5.
I20260430 07:59:06.215215 10011 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:06.216614 10003 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:06.231681 9904 leader_election.cc:403] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260430 07:59:06.221855 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:06.217897 11804 raft_consensus.cc:2804] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:59:06.238189 11804 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:06.238639 11882 raft_consensus.cc:3055] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:06.238838 11804 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:06.239131 11882 raft_consensus.cc:740] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:06.240132 11882 consensus_queue.cc:260] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.245915 10004 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260430 07:59:06.182512 10005 raft_consensus.cc:2955] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:06.176167 11769 consensus_queue.cc:1048] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
W20260430 07:59:06.224315 11929 log.cc:927] Time spent T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 1.422s user 0.005s sys 0.000s
I20260430 07:59:06.227609 11859 consensus_queue.cc:260] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.202147 11857 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:06.230620 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 61361.3us
I20260430 07:59:06.222971 11738 consensus_queue.cc:1243] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer 77e19ab653f540339c1c613fad1451b5 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260430 07:59:06.241957 11722 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:06.242954 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.243355 9977 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:06.248909 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.254227 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.256356 11882 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:59:06.257560 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:06.262310 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 63574.8us
I20260430 07:59:06.256253 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.256253 11601 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:06.273631 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:06.293715 9862 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:06.279188 9904 leader_election.cc:400] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Vote denied by peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679) with higher term. Message: Invalid argument: T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6.
I20260430 07:59:06.283016 11724 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:06.284801 9828 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:06.285905 10012 raft_consensus.cc:1240] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:06.286773 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.288978 10004 raft_consensus.cc:1275] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260430 07:59:06.289635 10008 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:06.289556 11879 consensus_queue.cc:260] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.296582 10003 raft_consensus.cc:1240] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:59:06.261171 11789 raft_consensus.cc:3055] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:06.304750 10007 raft_consensus.cc:2393] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:06.312482 9929 raft_consensus.cc:1240] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:06.303010 11712 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:06.306610 11789 raft_consensus.cc:740] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:06.315665 11712 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:06.323565 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 47657.5us
I20260430 07:59:06.324126 10011 raft_consensus.cc:1240] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 5. Current term is 6. Ops: []
W20260430 07:59:06.324399 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 107211us
I20260430 07:59:06.327134 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 216) took 2023 ms. Trace:
I20260430 07:59:06.327572 9854 rpcz_store.cc:276] 0430 07:59:04.304240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.307216 (+ 2976us) service_pool.cc:225] Handling call
0430 07:59:04.483091 (+175875us) raft_consensus.cc:1471] Updating replica for 3 ops
0430 07:59:04.554432 (+ 71341us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:04.554466 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0430 07:59:04.564940 (+ 10474us) log.cc:844] Serialized 616 byte log entry
0430 07:59:04.673852 (+108912us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:04.673920 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:04.674040 (+ 120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:06.189110 (+1515070us) spinlock_profiling.cc:243] Waited 35.9 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:06.323207 (+134097us) raft_consensus.cc:1685] finished
0430 07:59:06.323231 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:06.327065 (+ 3834us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17507,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":189,"spinlock_wait_cycles":82585472,"thread_start_us":108231,"threads_started":1,"wal-append.queue_time_us":221368}
W20260430 07:59:06.330164 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1334) took 2161 ms (client timeout 1871 ms). Trace:
W20260430 07:59:06.330641 9937 rpcz_store.cc:269] 0430 07:59:04.169228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.608601 (+439373us) service_pool.cc:225] Handling call
0430 07:59:06.329957 (+1721356us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540003c354. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:06.330090 (+ 133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3046755584}
I20260430 07:59:06.308085 9828 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:06.307125 11722 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:59:06.334589 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:06.334972 9828 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:06.334903 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:06.337988 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:06.338231 10006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1602) took 1472 ms. Trace:
I20260430 07:59:06.338662 10006 rpcz_store.cc:276] 0430 07:59:04.866079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:04.866315 (+ 236us) service_pool.cc:225] Handling call
0430 07:59:06.319824 (+1453509us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400097754. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:06.338183 (+ 18359us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2538657664}
I20260430 07:59:06.340776 9862 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:59:06.355259 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 54299.9us
W20260430 07:59:06.420224 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:06.423283 9978 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:06.427160 9904 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:06.312300 11972 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260430 07:59:06.462284 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:06.436600 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 96983.2us
W20260430 07:59:06.466147 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:06.436023 9860 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:06.436084 9828 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:06.436492 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 98991.7us
W20260430 07:59:06.471024 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:06.471864 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:06.471935 10008 raft_consensus.cc:1240] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 3. Current term is 4. Ops: []
W20260430 07:59:06.472812 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:06.469532 11841 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:06.431799 11797 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:06.351294 11974 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:06.462005 11712 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:06.427227 11859 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:06.480979 9828 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:06.496444 10006 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "77e19ab653f540339c1c613fad1451b5"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:06.495698 11625 raft_consensus.cc:3055] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:06.492480 11724 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:06.479135 11807 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:06.472738 11619 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:06.508150 9903 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:06.538578 11890 consensus_queue.cc:1048] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:59:06.511343 10006 raft_consensus.cc:606] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Received request to transfer leadership to TS 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:06.538913 11807 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.514277 11625 raft_consensus.cc:740] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:06.541258 10006 raft_consensus.cc:614] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:06.542254 11625 consensus_queue.cc:260] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.543926 11625 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:06.544185 11619 raft_consensus.cc:697] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:06.544870 9862 raft_consensus.cc:1275] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260430 07:59:06.545300 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:06.521471 11761 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:06.546020 11761 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.545956 11619 consensus_queue.cc:237] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.547492 11804 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.472919 11789 consensus_queue.cc:260] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.522459 11748 consensus_queue.cc:1059] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.025s
I20260430 07:59:06.523510 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:06.526539 11857 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:06.529762 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:06.530336 11952 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:06.514529 11974 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:06.538273 9854 raft_consensus.cc:1275] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:06.563462 9937 raft_consensus.cc:1240] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
W20260430 07:59:06.564482 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 239953us
I20260430 07:59:06.568432 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:06.570574 9904 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:06.572636 11827 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:06.518527 11802 consensus_queue.cc:1059] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
W20260430 07:59:06.574734 11976 log.cc:927] Time spent T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260430 07:59:06.575232 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:06.575917 9860 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:06.570736 11952 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.577910 11952 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:06.579852 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 69930.5us
I20260430 07:59:06.549365 11789 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:06.548533 11761 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:06.436666 11972 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:06.588133 10010 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:06.590958 10009 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:06.592001 10009 raft_consensus.cc:606] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:06.592518 10009 raft_consensus.cc:614] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:06.595589 9854 raft_consensus.cc:1275] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:06.589424 10534 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:06.567953 11974 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:06.590302 11972 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:06.559336 11712 consensus_queue.cc:1059] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
W20260430 07:59:06.624156 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 116829us
I20260430 07:59:06.600250 11974 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:06.630344 9855 raft_consensus.cc:1275] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:59:06.635025 10009 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:06.635699 10009 raft_consensus.cc:606] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:06.632961 9828 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:06.610555 11857 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.023s
I20260430 07:59:06.627784 11756 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:06.648128 11735 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20260430 07:59:06.649837 11735 consensus_queue.cc:1059] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:59:06.652475 11601 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679) with higher term. Message: Invalid argument: T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5.
I20260430 07:59:06.655658 11952 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:06.656150 11952 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:06.656518 11952 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:06.653399 11756 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:06.643015 11978 log.cc:927] Time spent T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.058s user 0.000s sys 0.002s
I20260430 07:59:06.667881 9977 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:06.667173 11712 consensus_queue.cc:1048] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.005s
I20260430 07:59:06.672349 11712 consensus_queue.cc:1243] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer 48e81934d52a4e93be5171ceea193032 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:59:06.676011 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:06.676702 9863 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:06.678351 9978 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:06.673954 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:06.666162 11756 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:06.668380 11804 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:06.688851 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 64821.5us
I20260430 07:59:06.680948 11857 consensus_queue.cc:1048] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260430 07:59:06.690744 11831 raft_consensus.cc:3055] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:06.701370 9828 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:06.707988 11623 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:06.710414 11756 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.711191 11889 raft_consensus.cc:3055] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:59:06.731963 11738 consensus_queue.cc:1059] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:06.734537 11889 raft_consensus.cc:740] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:06.735826 11889 consensus_queue.cc:260] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.739351 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:06.739781 11889 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:06.734730 11625 consensus_queue.cc:1059] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:06.732647 11756 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:06.740094 9863 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:06.740554 11336 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:06.747704 9977 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:06.732401 10533 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:06.777693 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 84846.6us
I20260430 07:59:06.806479 9828 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:06.700528 11735 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:06.811480 9860 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:06.822132 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 89767.2us
I20260430 07:59:06.715342 11807 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:06.700882 11831 raft_consensus.cc:740] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:06.836066 9978 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:06.839435 10008 raft_consensus.cc:3055] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:06.807934 11336 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:06.840936 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:06.842159 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 94624.7us
I20260430 07:59:06.811153 11761 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:06.846769 11336 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:06.852629 9977 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:06.841125 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:06.842872 10008 raft_consensus.cc:740] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:06.814083 11841 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:06.861692 10010 raft_consensus.cc:1217] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-5.5] Dedup: 1.3->[2.4-5.5]
I20260430 07:59:06.865396 11760 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:06.865911 11760 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.868662 11760 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:06.833189 11835 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260430 07:59:06.871244 11985 log.cc:927] Time spent T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20260430 07:59:06.874095 11986 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.012s
I20260430 07:59:06.791508 11974 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.885756 11974 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:06.858723 10008 consensus_queue.cc:260] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.851542 11722 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:06.886845 11756 raft_consensus.cc:3055] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:06.887359 11756 raft_consensus.cc:740] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:06.880337 11797 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:06.880951 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 74567.2us
I20260430 07:59:06.880528 11831 consensus_queue.cc:260] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.862555 10010 pending_rounds.cc:85] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 3
W20260430 07:59:06.883441 11882 consensus_meta.cc:287] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.592s user 0.004s sys 0.008s
I20260430 07:59:06.859632 11841 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:06.864962 11545 raft_consensus.cc:3055] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:06.886914 10008 raft_consensus.cc:3060] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:06.890677 10010 pending_rounds.cc:107] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20260430 07:59:06.896724 11831 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:06.898880 10010 raft_consensus.cc:2889] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:06.884023 11952 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:06.893774 11827 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:06.902240 9855 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:06.903615 11952 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:06.897742 11882 consensus_queue.cc:1175] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "27f97a330f374849be58a168318c1ec6" responder_term: 4 status { last_received { term: 4 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 4 index: 5 } }
I20260430 07:59:06.906632 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:06.911157 9862 raft_consensus.cc:1240] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
W20260430 07:59:06.908367 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 55798.8us
I20260430 07:59:06.895371 11807 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:06.898088 11841 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:06.912604 9977 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:06.897485 11545 raft_consensus.cc:740] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:06.921128 11545 consensus_queue.cc:260] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:06.924576 9863 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:06.925786 10534 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:59:06.947162 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:06.966053 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:06.977192 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 51495.9us
W20260430 07:59:06.980247 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 91623.8us
W20260430 07:59:06.983098 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 68894.6us
I20260430 07:59:06.922758 11545 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:06.989032 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:06.979936 11722 raft_consensus.cc:697] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:07.002027 9828 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:06.990375 11987 log.cc:927] Time spent T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260430 07:59:06.985584 11750 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:06.982640 11827 raft_consensus.cc:697] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:59:07.008814 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 96254us
I20260430 07:59:07.020294 10008 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 6 index: 8. (term mismatch)
I20260430 07:59:07.020347 9828 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:07.020895 10008 pending_rounds.cc:85] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 7
I20260430 07:59:07.021179 10008 pending_rounds.cc:107] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 5.8
I20260430 07:59:07.021464 10008 raft_consensus.cc:2889] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:07.021894 11797 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:06.916231 11756 consensus_queue.cc:260] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:07.022306 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:59:07.005868 11952 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:07.030839 9828 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:07.031616 9977 connection.cc:254] RPC call timeout handler was delayed by -2.073590164000052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1938s for a response
I20260430 07:59:07.000041 11850 consensus_queue.cc:1059] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.023s
I20260430 07:59:07.039736 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:07.041523 11952 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:07.042042 10564 raft_consensus.cc:1240] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:07.043242 11822 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.017s
I20260430 07:59:07.022985 11807 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:07.020117 11750 raft_consensus.cc:697] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
W20260430 07:59:07.032768 11859 consensus_meta.cc:287] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.530s user 0.014s sys 0.000s
W20260430 07:59:07.054296 9977 connection.cc:254] RPC call timeout handler was delayed by -1.5641729769999984s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
I20260430 07:59:07.022392 11797 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:07.034520 11974 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:07.070971 9828 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:59:07.079516 11997 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:07.080027 11997 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:07.080437 11997 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:07.080793 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:07.075901 11794 raft_consensus.cc:3055] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:07.086177 11794 raft_consensus.cc:740] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:07.087134 11794 consensus_queue.cc:260] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.088243 11794 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:07.092340 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.092454 11827 consensus_queue.cc:237] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.093113 11997 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:07.103128 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.100459 9856 raft_consensus.cc:1240] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:07.064930 11761 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:07.046600 11336 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:07.097904 11997 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:07.117489 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 51193.4us
I20260430 07:59:07.075623 11797 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:07.099792 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1318) took 2969 ms. Trace:
I20260430 07:59:07.120450 9977 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:07.098641 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:07.098425 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:07.077330 11974 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:07.122792 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.098547 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:07.129383 10003 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:07.116510 11748 consensus_queue.cc:1243] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer 48e81934d52a4e93be5171ceea193032 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:59:07.114569 11623 consensus_queue.cc:1059] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260430 07:59:07.099912 11972 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:07.136158 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 55522us
W20260430 07:59:07.137775 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.137866 9977 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:07.097656 11722 consensus_queue.cc:237] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.151152 11623 raft_consensus.cc:3055] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:59:07.151676 11623 raft_consensus.cc:740] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:07.131875 11974 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:07.129645 11859 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:07.125540 11616 consensus_queue.cc:1059] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260430 07:59:07.148319 11336 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:07.140151 11735 raft_consensus.cc:2749] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:07.125758 11761 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:07.173237 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:07.121922 11826 consensus_queue.cc:1059] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260430 07:59:07.156071 9862 raft_consensus.cc:1240] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:07.162191 9828 leader_election.cc:400] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:07.166319 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:07.167543 11616 raft_consensus.cc:3055] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:07.171074 11952 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:59:07.171913 11794 consensus_queue.cc:1059] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260430 07:59:07.124322 9930 rpcz_store.cc:276] 0430 07:59:04.130633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.007431 (+1876798us) service_pool.cc:225] Handling call
0430 07:59:06.007756 (+ 325us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:06.119566 (+111810us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:06.124722 (+ 5156us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:06.192442 (+ 67720us) log.cc:844] Serialized 26 byte log entry
0430 07:59:06.462081 (+269639us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:06.495478 (+ 33397us) log.cc:844] Serialized 14 byte log entry
0430 07:59:07.099094 (+603616us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b180016a740. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb566159e1a 00007fb56f7f25c9 00007fb5660181f0 00007fb5661e2b24 00007fb5661e2a95 00007fb56927b440 00007fb56928daf6 00007fb56929e667 00007fb56929e5e8
0430 07:59:07.099295 (+ 201us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:07.099393 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:07.099447 (+ 54us) raft_consensus.cc:1685] finished
0430 07:59:07.099564 (+ 117us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:07.099723 (+ 159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27337,"prepare.queue_time_us":332861,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":9638,"raft.queue_time_us":654212,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":7777,"spinlock_wait_cycles":87626112,"thread_start_us":98903,"threads_started":1,"wal-append.queue_time_us":278937}
I20260430 07:59:07.174854 10565 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:07.181758 9854 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:59:07.181789 11616 raft_consensus.cc:740] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:07.195851 11794 raft_consensus.cc:3055] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:07.196269 11794 raft_consensus.cc:740] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:07.196475 11616 consensus_queue.cc:260] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:07.197108 11794 consensus_queue.cc:260] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:07.198114 11616 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:07.198423 11794 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:07.152670 11623 consensus_queue.cc:260] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:07.185369 9863 raft_consensus.cc:1275] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:07.185727 11995 raft_consensus.cc:3055] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:07.201758 11995 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:07.202937 11995 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.204279 11995 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:07.204509 11804 raft_consensus.cc:3055] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
W20260430 07:59:07.204429 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 717374us
I20260430 07:59:07.205096 11804 raft_consensus.cc:740] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:07.101877 11750 consensus_queue.cc:237] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:07.155663 11972 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:07.164327 11619 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:07.239547 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:07.275621 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.281850 9978 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:07.287408 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.163947 11859 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.222469 11742 raft_consensus.cc:3055] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:07.210147 11789 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:07.293327 11769 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260430 07:59:07.316955 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 154914us
I20260430 07:59:07.318580 9828 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:07.328828 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:07.329653 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:07.330436 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.306668 11742 raft_consensus.cc:740] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:07.301780 11826 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:07.339046 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:07.341548 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:07.206188 11804 consensus_queue.cc:260] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.339530 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:07.351635 10008 raft_consensus.cc:2368] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6.
W20260430 07:59:07.355026 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.355145 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:07.358565 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:07.310236 11769 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:07.360289 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 793836us
I20260430 07:59:07.332504 11952 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:07.360230 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:07.373703 9931 raft_consensus.cc:1217] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Deduplicated request from leader. Original: 3.5->[5.6-5.6] Dedup: 5.6->[]
I20260430 07:59:07.376940 9931 raft_consensus.cc:1240] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260430 07:59:07.380801 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.382526 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:07.383462 10007 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 2.
W20260430 07:59:07.384765 11996 log.cc:927] Time spent T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260430 07:59:07.385182 9934 raft_consensus.cc:1240] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:07.360455 11742 consensus_queue.cc:260] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.392030 9930 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8"
dest_uuid: "77e19ab653f540339c1c613fad1451b5"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:52704
I20260430 07:59:07.393190 9930 raft_consensus.cc:606] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:07.393767 9930 raft_consensus.cc:614] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:07.407287 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:07.399847 11769 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:07.400733 9936 raft_consensus.cc:1240] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:07.401100 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:07.308351 11619 consensus_queue.cc:1048] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260430 07:59:07.398120 9929 raft_consensus.cc:1275] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260430 07:59:07.402380 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:07.415802 10004 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:07.411496 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:07.413592 9904 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:07.418823 11804 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:07.419148 11990 log.cc:927] Time spent T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260430 07:59:07.420226 11735 raft_consensus.cc:993] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: : Instructing follower 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:59:07.420734 11735 raft_consensus.cc:1081] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Signalling peer 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:59:07.421820 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:07.422901 11765 consensus_queue.cc:1059] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:07.424145 11765 raft_consensus.cc:3055] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:07.424665 11765 raft_consensus.cc:740] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:07.427240 9861 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c"
dest_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:37838
I20260430 07:59:07.428153 9861 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:59:07.428540 9861 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:07.436457 9861 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.437865 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:07.443329 11859 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:07.346583 11789 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:07.445333 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:07.448854 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 240880us
I20260430 07:59:07.448935 9863 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:59:07.449988 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:07.453406 9977 leader_election.cc:336] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.871s (SENT)
I20260430 07:59:07.453456 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:07.455047 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:07.454550 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 205) took 2457 ms. Trace:
W20260430 07:59:07.459334 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 45802us
I20260430 07:59:07.460662 9904 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:07.388867 11797 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.466933 9861 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:07.468063 9978 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:07.402303 11952 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:07.480502 11501 raft_consensus.cc:993] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: : Instructing follower 27f97a330f374849be58a168318c1ec6 to start an election
W20260430 07:59:07.504237 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 93187.1us
I20260430 07:59:07.474772 11972 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:07.482482 11748 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:07.483512 11891 raft_consensus.cc:1081] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Signalling peer 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:59:07.482591 11952 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:07.462198 11789 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.513851 11831 consensus_queue.cc:1048] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260430 07:59:07.509414 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 49053.6us
W20260430 07:59:07.516985 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 63592.1us
I20260430 07:59:07.515694 11849 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:07.489147 11997 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:07.516973 9861 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260430 07:59:07.577729 10005 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
W20260430 07:59:07.567987 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 361844us
I20260430 07:59:07.629665 9903 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:07.607326 9904 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:07.584785 10006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:44006 (request call id 530) took 1033 ms. Trace:
W20260430 07:59:07.567380 11993 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260430 07:59:07.614523 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 206142us
W20260430 07:59:07.615789 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:07.607363 9977 leader_election.cc:336] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.066s (SENT)
I20260430 07:59:07.630905 9903 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:07.645308 9903 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:07.581987 11756 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:07.578243 11761 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:59:07.609185 11841 consensus_meta.cc:287] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.626s user 0.011s sys 0.003s
I20260430 07:59:07.643365 9977 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:07.632529 10006 rpcz_store.cc:276] 0430 07:59:06.551723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.551953 (+ 230us) service_pool.cc:225] Handling call
0430 07:59:07.584221 (+1032268us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b54000cd354. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ed470 00007fb56928c31a 00007fb56e25b11e 00007fb5677d7437 00007fb5677d73c2 00007fb5677d72f1 00007fb5677d726c 00007fb5677d6522 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:07.584721 (+ 500us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1030618880}
I20260430 07:59:07.517398 11972 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:07.649870 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.650287 10005 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:07.650931 10005 raft_consensus.cc:606] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:07.651283 10005 raft_consensus.cc:614] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:07.589910 12000 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:07.654959 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:07.656740 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:07.505332 11765 consensus_queue.cc:260] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.686496 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:07.688755 10004 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260430 07:59:07.689107 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:07.689754 10011 raft_consensus.cc:3055] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
W20260430 07:59:07.675900 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:07.693545 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 86162.5us
W20260430 07:59:07.694265 9977 leader_election.cc:336] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.067s (SENT)
I20260430 07:59:07.646582 11997 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.661785 11841 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.646317 11850 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:07.690349 10011 raft_consensus.cc:740] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:59:07.706565 9977 leader_election.cc:336] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.068s (SENT)
I20260430 07:59:07.612301 11974 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:07.646263 12004 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:07.715137 10003 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:07.716552 9860 raft_consensus.cc:1240] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:07.720300 10011 consensus_queue.cc:260] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.649354 11760 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:07.730914 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 123619us
I20260430 07:59:07.698897 11849 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:07.748656 10006 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260430 07:59:07.709537 12006 log.cc:927] Time spent T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20260430 07:59:07.747531 10011 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:07.764427 11797 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:07.784066 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 95981.8us
I20260430 07:59:07.721083 11794 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:07.708839 11972 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:07.798686 9859 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:07.690743 11748 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.071s
W20260430 07:59:07.795812 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 89240.2us
I20260430 07:59:07.801820 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:07.748776 11997 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:07.804004 10006 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:07.805653 12020 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:07.807004 11841 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:07.709699 12009 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:07.699589 12002 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:07.711594 11807 consensus_queue.cc:1059] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.100s
I20260430 07:59:07.811295 11974 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:07.709760 12011 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:07.816251 9977 leader_election.cc:336] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.069s (SENT)
W20260430 07:59:07.799692 9828 leader_election.cc:336] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.969s (SENT)
I20260430 07:59:07.802827 12003 consensus_queue.cc:1048] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.157s
I20260430 07:59:07.802842 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
W20260430 07:59:07.828356 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.825793 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:07.782999 11765 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:07.685918 12000 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.835038 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:07.880570 9937 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:07.838264 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.834743 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:07.839246 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:07.839850 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:07.801860 11849 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260430 07:59:07.832602 12009 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:07.782406 11760 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:07.843582 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:07.912995 9932 raft_consensus.cc:2376] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5. Re-sending same reply.
W20260430 07:59:07.913362 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 150950us
I20260430 07:59:07.884990 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260430 07:59:07.885507 12008 log.cc:927] Time spent T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.273s user 0.002s sys 0.000s
I20260430 07:59:07.885233 9758 catalog_manager.cc:5671] T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 3 to 4, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 4 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:59:07.886229 11724 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:07.887039 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.887969 11742 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:07.896054 11616 consensus_meta.cc:287] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.697s user 0.004s sys 0.005s
W20260430 07:59:07.897776 12007 log.cc:927] Time spent T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.322s user 0.000s sys 0.002s
I20260430 07:59:07.933636 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:07.934213 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:07.878909 9931 raft_consensus.cc:2368] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:07.934934 9938 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:59:07.935734 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:07.908841 10561 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260430 07:59:07.846196 9936 raft_consensus.cc:1240] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260430 07:59:07.882131 9933 raft_consensus.cc:2376] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6. Re-sending same reply.
I20260430 07:59:07.914695 9904 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:07.916648 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.916009 9828 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:59:07.842401 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 179967us
I20260430 07:59:07.909943 12003 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:07.836102 12002 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:07.922520 11760 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:07.902155 12011 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:07.927654 12000 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:07.953060 11841 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:07.943050 12002 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:07.954084 10562 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
W20260430 07:59:07.954620 9747 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 48e81934d52a4e93be5171ceea193032 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:07.882447 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:07.933036 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.943801 9904 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:07.959985 11802 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:07.942168 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:07.961762 9931 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:07.912056 12009 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:07.969367 10565 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:07.947131 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:07.970072 9904 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:07.948258 12011 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:07.918895 11789 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:08.022010 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:07.950397 11804 consensus_meta.cc:287] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.531s user 0.006s sys 0.005s
I20260430 07:59:07.943148 11601 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:07.952667 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 166843us
I20260430 07:59:07.952747 11619 raft_consensus.cc:3055] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
W20260430 07:59:08.033303 9828 leader_election.cc:336] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.230s (SENT)
I20260430 07:59:07.954953 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 186) took 1119 ms. Trace:
I20260430 07:59:08.033780 9828 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:07.958212 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:07.931555 11769 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:07.944125 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 200158us
I20260430 07:59:07.963130 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:07.968176 9937 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:08.033840 10010 rpcz_store.cc:276] 0430 07:59:06.835123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.835248 (+ 125us) service_pool.cc:225] Handling call
0430 07:59:06.861407 (+ 26159us) raft_consensus.cc:1471] Updating replica for 3 ops
0430 07:59:06.984413 (+123006us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:59:06.984458 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:07.188918 (+204460us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b2c0008b928. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb567b7dc81 00007fb567b8d5e2 00007fb5692bede1 00007fb56928381a 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22
0430 07:59:07.189651 (+ 733us) log.cc:844] Serialized 52 byte log entry
0430 07:59:07.315738 (+126087us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:07.315775 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:07.315841 (+ 66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:07.954625 (+638784us) raft_consensus.cc:1685] finished
0430 07:59:07.954648 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:07.954883 (+ 235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79751,"raft.queue_time_us":74711,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":7645,"spinlock_wait_cycles":2393472,"thread_start_us":125683,"threads_started":1,"wal-append.queue_time_us":297587}
W20260430 07:59:07.919243 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 228953us
I20260430 07:59:07.939638 11724 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:07.946280 10559 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:07.455814 9935 rpcz_store.cc:276] 0430 07:59:04.997411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:06.052024 (+1054613us) service_pool.cc:225] Handling call
0430 07:59:06.116941 (+ 64917us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:06.117135 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:06.260930 (+143795us) log.cc:844] Serialized 14 byte log entry
0430 07:59:06.783304 (+522374us) spinlock_profiling.cc:243] Waited 4.51 ms on lock 0x7b180012c040. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb566159e1a 00007fb56f7f25c9 00007fb5660181f0 00007fb5661e2b24 00007fb5661e2a95 00007fb56cc86837 00007fb56928dd87 00007fb56929d49f 00007fb56929d418
0430 07:59:06.783710 (+ 406us) log.cc:844] Serialized 14 byte log entry
0430 07:59:07.444875 (+661165us) log.cc:844] Serialized 14 byte log entry
0430 07:59:07.445298 (+ 423us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:07.445359 (+ 61us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:07.445411 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:07.445480 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:07.454491 (+ 9011us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":500957,"prepare.queue_time_us":343733,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":67,"raft.queue_time_us":593544,"raft.run_cpu_time_us":735,"raft.run_wall_time_us":27151,"spinlock_wait_cycles":10379136,"thread_start_us":385464,"threads_started":2,"wal-append.queue_time_us":267441}
I20260430 07:59:07.971848 10011 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:07.973784 9933 raft_consensus.cc:2393] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:07.982640 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:08.022682 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 52692.8us
I20260430 07:59:07.944682 11616 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:08.030627 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 214341us
W20260430 07:59:08.030800 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 69028.8us
I20260430 07:59:07.833945 12020 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:07.981511 11995 consensus_meta.cc:287] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.764s user 0.009s sys 0.005s
I20260430 07:59:08.044763 11841 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:08.094584 9977 leader_election.cc:336] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.199s (SENT)
I20260430 07:59:08.094956 11841 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.095531 9933 pending_rounds.cc:85] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 2
I20260430 07:59:08.095870 9933 pending_rounds.cc:107] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260430 07:59:08.096118 9933 raft_consensus.cc:2889] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:08.096341 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:07.944936 12003 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.096962 9936 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:08.106263 9904 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:59:08.126623 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:08.092137 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 58862.2us
I20260430 07:59:08.030452 11802 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:08.099884 9859 raft_consensus.cc:1275] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260430 07:59:08.124781 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:59:08.139204 9859 pending_rounds.cc:85] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 1
I20260430 07:59:08.139559 9859 pending_rounds.cc:107] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260430 07:59:08.033596 11619 raft_consensus.cc:740] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:08.139801 9859 raft_consensus.cc:2889] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:08.110692 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:08.141070 9863 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:08.132560 9857 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260430 07:59:08.142207 9863 raft_consensus.cc:1240] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:08.072737 9931 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:08.046126 9937 raft_consensus.cc:1240] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:08.047914 9860 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742"
dest_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:37838
I20260430 07:59:08.149096 10012 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
W20260430 07:59:08.059926 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:08.147011 9758 catalog_manager.cc:5671] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 3 to 4, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 4 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:59:08.149420 9860 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:59:08.151955 9860 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:08.152748 9932 raft_consensus.cc:2410] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260430 07:59:08.043797 12000 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:59:08.157720 9977 leader_election.cc:336] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.901s (SENT)
I20260430 07:59:08.157734 9828 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:08.141403 9855 raft_consensus.cc:2955] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:08.112694 11831 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:08.171916 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:08.172024 9977 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:08.097070 11841 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:08.097496 11616 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:08.093261 11769 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:08.178177 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 72009.4us
I20260430 07:59:08.161437 11849 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:08.180330 10004 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260430 07:59:08.174072 9828 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:08.183897 9904 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:08.136216 12004 raft_consensus.cc:697] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:08.101445 11807 consensus_queue.cc:1059] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260430 07:59:08.185982 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:59:08.046255 11760 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.131s
I20260430 07:59:08.148006 12021 consensus_queue.cc:1059] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260430 07:59:08.181187 11974 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260430 07:59:08.236737 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:08.237898 9904 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:08.189066 11889 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:08.189381 12025 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:08.202231 9828 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:08.252106 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 94376.2us
W20260430 07:59:08.288956 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:08.248222 9860 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.180779 11997 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:08.298029 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 95863.7us
W20260430 07:59:08.313290 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:08.330472 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 157569us
W20260430 07:59:08.330967 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 101534us
I20260430 07:59:08.332617 9978 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:08.338126 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 111405us
I20260430 07:59:08.337997 9758 catalog_manager.cc:5671] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 3 to 4, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 4 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:59:08.339964 9903 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:08.225385 11769 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:08.202013 12000 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:08.182950 11831 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:08.342932 9860 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:08.345577 9903 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:08.329737 9828 leader_election.cc:400] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6.
I20260430 07:59:08.222069 11849 raft_consensus.cc:697] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:08.229110 11974 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:08.302910 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 65107.9us
I20260430 07:59:08.361569 9828 leader_election.cc:403] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260430 07:59:08.361562 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 170421us
W20260430 07:59:08.364935 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:08.371999 9904 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:08.337985 11841 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:08.252238 11807 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:08.329209 11997 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:08.250753 11765 consensus_queue.cc:1048] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.121s
I20260430 07:59:08.397006 9904 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:08.347391 11601 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:08.448742 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 116221us
W20260430 07:59:08.410318 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:08.396903 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 51636.8us
I20260430 07:59:08.251919 11802 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.126s
W20260430 07:59:08.457185 11952 consensus_meta.cc:287] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.888s user 0.007s sys 0.007s
I20260430 07:59:08.375623 12028 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:59:08.457695 11952 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.422878 9904 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:08.355353 12033 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:08.376121 11974 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:08.351588 11889 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.359932 12011 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:08.452426 9758 catalog_manager.cc:5671] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 1 to 4, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 4 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:59:08.466414 9903 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:08.397656 11891 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:08.250541 11619 consensus_queue.cc:260] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.347050 12025 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.421518 11997 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:08.467893 12027 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.429s user 0.000s sys 0.001s
W20260430 07:59:08.497815 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 75397.4us
W20260430 07:59:08.501611 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:08.522774 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 146600us
W20260430 07:59:08.487772 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 157975us
I20260430 07:59:08.468668 11724 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:08.465090 12022 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:08.421794 11807 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:08.546029 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 79651.1us
I20260430 07:59:08.550334 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:08.497151 12003 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:08.556715 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 98992.8us
I20260430 07:59:08.550999 9828 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:08.526856 11601 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:08.522794 12031 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:08.563171 9903 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:08.534020 11891 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:08.558810 9758 catalog_manager.cc:5671] T 2e8dad8acea649479d1e927cd32599fe P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 2 to 3, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 3 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:59:08.577054 9860 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:59:08.418603 12004 consensus_queue.cc:237] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:08.473394 12039 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.089s
I20260430 07:59:08.546705 12033 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:08.542640 12020 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:08.603598 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 238097us
I20260430 07:59:08.546521 11616 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:08.533813 11520 consensus_queue.cc:1048] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260430 07:59:08.552536 12000 raft_consensus.cc:697] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:08.611539 12000 consensus_queue.cc:237] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.529253 12011 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.616714 12011 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:08.597035 11879 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:08.546204 12044 consensus_queue.cc:1059] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260430 07:59:08.650445 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:08.596537 11891 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:08.603102 12003 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:59:08.656967 9759 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48090 (request call id 190) took 16308 ms (client timeout 15000 ms). Trace:
I20260430 07:59:08.634217 11601 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:08.678772 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:08.675480 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:08.633116 11769 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:08.580441 11804 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
W20260430 07:59:08.671223 9936 consensus_meta.cc:287] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.540s user 0.013s sys 0.000s
I20260430 07:59:08.685824 9936 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:08.687266 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:08.684672 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:08.635035 12031 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:08.677309 10008 raft_consensus.cc:2955] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:08.645010 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:08.695279 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:08.702513 10010 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 4: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:08.680712 10006 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
W20260430 07:59:08.699481 9855 consensus_meta.cc:287] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.527s user 0.013s sys 0.000s
W20260430 07:59:08.702680 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:08.556699 11849 consensus_queue.cc:237] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.720786 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:08.726240 10010 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
W20260430 07:59:08.728264 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 121918us
I20260430 07:59:08.732784 9860 raft_consensus.cc:1275] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260430 07:59:08.733955 9856 raft_consensus.cc:1240] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:08.735859 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:08.744405 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 193433us
I20260430 07:59:08.757505 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:08.673746 11831 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:08.761441 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:08.768282 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:08.769819 10009 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:08.772372 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:08.700397 12042 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:08.700353 12003 raft_consensus.cc:697] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:08.704407 11841 raft_consensus.cc:697] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:08.789515 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:08.793315 9828 leader_election.cc:336] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.771s (SENT)
I20260430 07:59:08.817086 9828 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:08.822105 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:08.828367 10533 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:08.837905 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:08.838893 10010 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:08.840341 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:08.842064 10533 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:59:08.844729 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:08.815140 12002 consensus_meta.cc:287] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.861s user 0.012s sys 0.000s
I20260430 07:59:08.832571 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:08.669592 12047 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.124s
I20260430 07:59:08.862262 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:08.700735 11601 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:08.863341 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 70079.2us
W20260430 07:59:08.876406 9828 leader_election.cc:336] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.814s (SENT)
I20260430 07:59:08.815403 11952 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:08.876916 9828 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:08.877636 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:08.833745 12022 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:08.894333 9856 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
W20260430 07:59:08.862442 12008 log.cc:927] Time spent T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20260430 07:59:08.890558 9858 raft_consensus.cc:1240] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
W20260430 07:59:08.907701 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 49103.9us
I20260430 07:59:08.914403 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:08.921530 10005 raft_consensus.cc:3055] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:08.922034 10005 raft_consensus.cc:740] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:08.922971 10005 consensus_queue.cc:260] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.861169 11889 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:08.874925 11616 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:08.916973 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:08.928630 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 52272.4us
I20260430 07:59:08.843530 11738 raft_consensus.cc:2804] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260430 07:59:08.931550 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:08.932979 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:08.937114 12037 log.cc:927] Time spent T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.391s user 0.000s sys 0.001s
I20260430 07:59:08.937413 11601 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:08.937763 12041 log.cc:927] Time spent T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.355s user 0.001s sys 0.000s
I20260430 07:59:08.864320 12050 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6.
I20260430 07:59:08.719638 12031 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:08.929953 11952 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:08.931598 10005 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:08.944074 9828 leader_election.cc:336] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.848s (SENT)
I20260430 07:59:08.975023 9828 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:08.925704 11807 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:08.979781 9978 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:08.870249 12002 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:08.970758 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:08.925398 11794 consensus_queue.cc:1048] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260430 07:59:08.984388 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:08.990267 9936 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:08.992128 10533 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:08.940778 12053 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:08.945369 12025 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:08.823254 12042 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:08.986109 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:08.952751 11738 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:09.010532 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.012048 9936 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:08.943776 11616 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:08.966388 12065 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:09.047895 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.049484 9858 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:08.997466 12053 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:08.913196 12022 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:08.998335 12054 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260430 07:59:08.670066 9759 rpcz_store.cc:269] 0430 07:58:52.055209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:52.060142 (+ 4933us) service_pool.cc:225] Handling call
0430 07:59:08.354142 (+16294000us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:58:55.971332 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:58:55.971969 (+ 637us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:58:55.972014 (+ 45us) write_op.cc:435] Acquired schema lock
0430 07:58:55.972032 (+ 18us) tablet.cc:662] Decoding operations
0430 07:58:55.975323 (+ 3291us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:58:55.975383 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0430 07:58:55.975430 (+ 47us) tablet.cc:685] Acquiring locks for 4 operations
0430 07:58:55.975833 (+ 403us) tablet.cc:701] Row locks acquired
0430 07:58:55.975849 (+ 16us) write_op.cc:260] PREPARE: finished
0430 07:58:55.976079 (+ 230us) write_op.cc:270] Start()
0430 07:58:55.976187 (+ 108us) write_op.cc:276] Timestamp: P: 1777535935976030 usec, L: 0
0430 07:58:55.976208 (+ 21us) op_driver.cc:348] REPLICATION: starting
0430 07:58:55.976943 (+ 735us) log.cc:844] Serialized 1695 byte log entry
0430 07:58:56.461979 (+485036us) op_driver.cc:464] REPLICATION: finished
0430 07:58:56.586166 (+124187us) write_op.cc:301] APPLY: starting
0430 07:58:56.586280 (+ 114us) tablet.cc:1367] starting BulkCheckPresence
0430 07:58:56.586476 (+ 196us) tablet.cc:1370] finished BulkCheckPresence
0430 07:58:56.863472 (+276996us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:58:56.945005 (+ 81533us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:58:56.945265 (+ 260us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0430 07:58:56.951390 (+ 6125us) write_op.cc:312] APPLY: finished
0430 07:58:56.975594 (+ 24204us) log.cc:844] Serialized 40 byte log entry
0430 07:58:56.975795 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:56.978129 (+ 2334us) write_op.cc:454] Released schema lock
0430 07:58:56.978184 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0430 07:58:56.980245 (+ 2061us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:58:56.980291 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":337504,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":63791,"apply.run_cpu_time_us":4256,"apply.run_wall_time_us":393392,"num_ops":4,"prepare.queue_time_us":734014,"prepare.run_cpu_time_us":31253,"prepare.run_wall_time_us":133817,"raft.queue_time_us":176255,"raft.run_cpu_time_us":5465,"raft.run_wall_time_us":52580,"replication_time_us":485563,"thread_start_us":518055,"threads_started":4,"wal-append.queue_time_us":296992,"wal-append.run_cpu_time_us":24298,"wal-append.run_wall_time_us":1731129}]]}
W20260430 07:59:09.052585 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 108501us
I20260430 07:59:08.998122 12050 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:08.952214 11841 consensus_queue.cc:237] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:09.009634 12062 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260430 07:59:08.978904 11952 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:09.058547 9828 leader_election.cc:336] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.850s (SENT)
I20260430 07:59:09.061936 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.010748 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:09.064069 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1372) took 1146 ms. Trace:
I20260430 07:59:09.064453 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1369) took 1160 ms. Trace:
I20260430 07:59:09.085520 9938 rpcz_store.cc:276] 0430 07:59:07.904513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:08.133200 (+228687us) service_pool.cc:225] Handling call
0430 07:59:08.160618 (+ 27418us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:08.174726 (+ 14108us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:08.174756 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:08.184866 (+ 10110us) log.cc:844] Serialized 26 byte log entry
0430 07:59:08.399232 (+214366us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:08.607322 (+208090us) log.cc:844] Serialized 14 byte log entry
0430 07:59:08.626923 (+ 19601us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:08.627029 (+ 106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:09.064244 (+437215us) raft_consensus.cc:1685] finished
0430 07:59:09.064266 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:09.064382 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82780,"prepare.queue_time_us":507661,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":34533,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":22009,"thread_start_us":292892,"threads_started":2,"wal-append.queue_time_us":324032}
I20260430 07:59:09.065357 9978 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:09.068512 9828 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:09.081173 12064 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.010946 11738 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:09.084343 9937 rpcz_store.cc:276] 0430 07:59:07.917790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:08.145134 (+227344us) service_pool.cc:225] Handling call
0430 07:59:08.162942 (+ 17808us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:08.163133 (+ 191us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:08.163166 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:08.169702 (+ 6536us) log.cc:844] Serialized 26 byte log entry
0430 07:59:08.335439 (+165737us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:08.612279 (+276840us) log.cc:844] Serialized 14 byte log entry
0430 07:59:08.628085 (+ 15806us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:08.628301 (+ 216us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:09.063864 (+435563us) raft_consensus.cc:1685] finished
0430 07:59:09.063890 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:09.064016 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47978,"prepare.queue_time_us":506307,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":63,"raft.queue_time_us":32215,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":12809,"thread_start_us":294370,"threads_started":2,"wal-append.queue_time_us":278920}
I20260430 07:59:09.010521 11807 consensus_queue.cc:1059] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260430 07:59:09.061000 12054 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:09.060484 12053 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.119220 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.004109 12051 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:09.111927 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:08.970242 12003 consensus_queue.cc:237] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:09.110409 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 51976.2us
W20260430 07:59:09.133894 9828 leader_election.cc:336] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.609s (SENT)
I20260430 07:59:09.063150 12060 raft_consensus.cc:2804] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
W20260430 07:59:09.126967 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:09.099186 11655 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:09.146238 11655 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:09.146778 11655 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.147336 11616 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:09.154307 11742 consensus_meta.cc:287] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 1.222s user 0.009s sys 0.005s
W20260430 07:59:09.156612 9826 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.011s sys 0.038s
W20260430 07:59:09.157089 9826 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.012s sys 0.038s
I20260430 07:59:09.161725 12042 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:09.162716 11997 consensus_meta.cc:287] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.605s user 0.011s sys 0.004s
I20260430 07:59:09.163216 11997 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.163597 11952 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.165763 11997 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:09.165943 12041 log.cc:927] Time spent T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260430 07:59:09.166739 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.167352 9938 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:09.168286 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.169368 11655 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.167121 12053 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.170409 12022 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.170923 12051 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.172179 12053 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:09.172725 11879 consensus_meta.cc:287] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.517s user 0.010s sys 0.000s
I20260430 07:59:09.174172 11655 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:09.174259 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 57428.6us
I20260430 07:59:09.175853 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.177045 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.177335 11724 consensus_queue.cc:1059] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:59:09.178059 11724 consensus_queue.cc:1243] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer 48e81934d52a4e93be5171ceea193032 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:59:09.178119 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.179492 9828 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:09.175197 9929 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:09.181026 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.182132 9938 raft_consensus.cc:2410] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:09.188921 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:09.189706 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:09.190430 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:09.191215 10008 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.204943 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.179934 9903 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:09.205632 10005 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:09.111699 12054 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:09.218999 12076 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:09.220016 11918 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:09.220594 11918 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:09.221239 11918 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.225571 12077 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.226419 12077 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.229508 12080 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:09.230697 12015 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.231403 12015 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.233031 12063 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.233832 12063 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:09.237517 12069 log.cc:927] Time spent T 7f111c6c48d34a6fb38048804967ed93 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20260430 07:59:09.239277 12078 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:09.239284 12072 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260430 07:59:09.239560 12077 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.236994 12063 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.246366 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:09.247162 10005 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:09.248585 10533 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:08.890328 11712 raft_consensus.cc:2955] T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
W20260430 07:59:09.255374 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 75473us
I20260430 07:59:09.114540 12050 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:09.173085 12051 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.258447 9860 raft_consensus.cc:1217] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.2->[4.3-4.3] Dedup: 4.3->[]
I20260430 07:59:09.260160 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.105374 12064 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.265046 12077 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:09.538990 9860 raft_consensus.cc:1240] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 6. Ops: [4.3-4.3]
I20260430 07:59:09.539319 12077 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:09.539927 12077 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:09.540277 9904 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:09.540534 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.541913 11804 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:09.533922 12074 log.cc:927] Time spent T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.281s user 0.000s sys 0.000s
W20260430 07:59:09.380400 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:09.362661 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:09.546803 9863 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:59:09.344461 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:09.543921 10008 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:09.378682 9857 raft_consensus.cc:1240] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260430 07:59:09.522617 12012 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:09.543190 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 409184us
W20260430 07:59:09.550298 9828 leader_election.cc:336] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.646s (SENT)
I20260430 07:59:09.550679 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.551007 11742 consensus_queue.cc:1059] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:59:09.551620 9857 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:09.552192 11804 raft_consensus.cc:3055] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:09.552667 11804 raft_consensus.cc:740] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:09.553521 11804 consensus_queue.cc:260] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.550680 9828 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:09.302189 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:09.276649 9977 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.011s sys 0.021s
W20260430 07:59:09.285558 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:09.336330 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:09.286695 10009 consensus_meta.cc:287] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.505s user 0.015s sys 0.000s
I20260430 07:59:09.543857 9938 raft_consensus.cc:2393] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:09.216336 12060 raft_consensus.cc:697] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:09.252091 11974 consensus_meta.cc:287] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.728s user 0.014s sys 0.000s
I20260430 07:59:09.556314 9855 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.554591 12077 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.542032 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:09.178572 12033 raft_consensus.cc:3055] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:09.549383 12012 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.433836 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.558104 10533 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:09.585611 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:09.310478 12051 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.557317 9904 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:09.555763 9977 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.949s user 0.017s sys 0.036s
I20260430 07:59:09.556489 10009 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
W20260430 07:59:09.386346 9930 consensus_meta.cc:287] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 1.264s user 0.010s sys 0.010s
I20260430 07:59:09.569772 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:09.538601 12001 log.cc:927] Time spent T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 2.033s user 0.005s sys 0.007s
I20260430 07:59:09.574689 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:09.574707 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:09.574988 11724 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.544703 11623 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:09.576359 9903 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:09.576339 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:09.537477 11891 consensus_meta.cc:287] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.858s user 0.016s sys 0.004s
W20260430 07:59:09.367954 12000 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s user 0.015s sys 0.019s
I20260430 07:59:09.581007 9863 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:59:09.470835 12009 consensus_meta.cc:287] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 1.433s user 0.013s sys 0.002s
I20260430 07:59:09.271435 11655 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:09.270679 11794 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:09.382782 12002 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.585068 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:09.434631 9854 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.068s sys 0.077s
I20260430 07:59:09.587172 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:09.634478 9854 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.860s user 0.090s sys 0.079s
I20260430 07:59:09.634711 9856 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
W20260430 07:59:09.586076 10012 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 983) took 2174 ms (client timeout 1989 ms). Trace:
W20260430 07:59:09.459545 12011 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.000s sys 0.015s
W20260430 07:59:09.635515 10012 rpcz_store.cc:269] 0430 07:59:07.412086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:07.412242 (+ 156us) service_pool.cc:225] Handling call
0430 07:59:09.585919 (+2173677us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b54003a0a54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:09.586019 (+ 100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1569811200}
W20260430 07:59:09.635982 12011 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.826s user 0.001s sys 0.017s
I20260430 07:59:09.272909 12022 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:09.591598 10005 raft_consensus.cc:1240] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:09.591816 9930 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:09.590621 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.592387 9904 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:09.378362 12031 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:09.599507 10004 raft_consensus.cc:2393] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:09.336308 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:09.645426 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 53088.6us
I20260430 07:59:09.555722 12079 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:09.647341 11802 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:09.647915 12012 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.650061 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:09.650915 10004 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:59:09.652017 10533 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:09.649991 12080 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:09.581084 12004 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:09.648679 9904 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:09.652840 12080 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.653540 12004 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.654012 11802 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:09.655230 9904 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:09.655597 12004 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.655894 12080 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:09.606994 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.584471 12065 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.656780 12065 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.657742 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.658496 9863 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:09.659493 10564 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:59:09.660198 12013 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.660751 12013 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:09.608709 12081 log.cc:927] Time spent T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20260430 07:59:09.637637 11655 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.602468 12078 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:09.663024 12078 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.604931 11724 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.609738 11891 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:09.609776 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:09.610946 12087 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:09.612246 11742 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:09.613258 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:09.614867 9828 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:09.617138 9932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1362) took 1790 ms (client timeout 2071 ms). Trace:
I20260430 07:59:09.613499 11804 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:09.601107 12051 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:09.619231 10565 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54778 (request call id 110) took 2233 ms (client timeout 1841 ms). Trace:
W20260430 07:59:09.622164 10559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54754 (request call id 267) took 2590 ms (client timeout 3108 ms). Trace:
I20260430 07:59:09.625569 11903 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:59:09.629222 9934 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1357) took 1804 ms (client timeout 1645 ms). Trace:
W20260430 07:59:09.689767 9934 rpcz_store.cc:269] 0430 07:59:07.824873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:07.825006 (+ 133us) service_pool.cc:225] Handling call
0430 07:59:09.628983 (+1803977us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54000b45d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:09.629142 (+ 159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2665520000}
I20260430 07:59:09.631022 11850 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:09.635063 9855 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:09.334908 12050 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.606866 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.585215 12033 raft_consensus.cc:740] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:09.668401 9861 raft_consensus.cc:1240] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:09.577976 11974 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.646690 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.570690 11849 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:09.674252 9859 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260430 07:59:09.664623 11623 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:09.694870 11849 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.646703 9938 raft_consensus.cc:2393] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:09.696054 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 946) took 1009 ms. Trace:
I20260430 07:59:09.655997 11802 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:09.671286 9904 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:59:09.674908 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:09.681999 10565 rpcz_store.cc:269] 0430 07:59:07.382912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:07.383142 (+ 230us) service_pool.cc:225] Handling call
0430 07:59:09.616358 (+2233216us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b54005b2f54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:09.616476 (+ 118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1199466880}
W20260430 07:59:09.675721 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 60923.6us
W20260430 07:59:09.677520 9932 rpcz_store.cc:269] 0430 07:59:07.827071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:07.935022 (+107951us) service_pool.cc:225] Handling call
0430 07:59:09.616957 (+1681935us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b5400347e54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:09.617080 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":261450624}
I20260430 07:59:09.639393 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:09.692719 9937 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:09.691900 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:09.701031 9828 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:09.701095 9858 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.696714 9938 rpcz_store.cc:276] 0430 07:59:08.686310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.575200 (+888890us) service_pool.cc:225] Handling call
0430 07:59:09.695989 (+120789us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:09.695209 11623 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:09.709519 9861 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:09.707068 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.713920 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.708359 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:09.699640 11903 raft_consensus.cc:697] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:09.716235 11623 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.708866 11742 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:09.716742 12033 consensus_queue.cc:260] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.717070 11903 consensus_queue.cc:237] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.717499 11742 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:09.717933 12033 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.718322 11742 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:09.720366 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.721097 9934 raft_consensus.cc:2368] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
W20260430 07:59:09.683243 12032 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 1.160s user 0.002s sys 0.000s
I20260430 07:59:09.721257 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.721509 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 952) took 1033 ms. Trace:
I20260430 07:59:09.721900 9934 rpcz_store.cc:276] 0430 07:59:08.688242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.690703 (+1002461us) service_pool.cc:225] Handling call
0430 07:59:09.721454 (+ 30751us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:09.722416 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.722765 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.640002 12022 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:09.675964 12080 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:09.748875 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.673877 12087 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:09.753489 9863 raft_consensus.cc:1240] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:59:09.723884 9934 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:09.745455 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:09.685482 10559 rpcz_store.cc:269] 0430 07:59:07.032137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:07.034252 (+ 2115us) service_pool.cc:225] Handling call
0430 07:59:09.621968 (+2587716us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b5400117e54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:09.622089 (+ 121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1880123648}
I20260430 07:59:09.747032 12002 consensus_queue.cc:1059] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:09.747977 11802 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.663883 11655 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:09.659521 12065 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.673266 11891 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:09.694893 11850 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.751322 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.755220 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 953) took 1066 ms. Trace:
I20260430 07:59:09.781663 9934 rpcz_store.cc:276] 0430 07:59:08.710197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.722509 (+1012312us) service_pool.cc:225] Handling call
0430 07:59:09.755109 (+ 32600us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:09.744942 9857 raft_consensus.cc:2410] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:09.755424 9937 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.755574 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.778033 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1480) took 1046 ms. Trace:
I20260430 07:59:09.696295 11952 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:09.680987 12051 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:09.639230 11794 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.764904 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1624) took 2100 ms. Trace:
I20260430 07:59:09.706223 12079 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:09.633558 12009 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.796419 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.750818 12022 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:09.783267 9855 raft_consensus.cc:2393] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 4: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:09.671185 11724 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.784039 9854 rpcz_store.cc:276] 0430 07:59:08.731401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:08.731527 (+ 126us) service_pool.cc:225] Handling call
0430 07:59:08.731872 (+ 345us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:08.732105 (+ 233us) raft_consensus.cc:1534] Early marking committed up to index 3
0430 07:59:09.634911 (+902806us) log.cc:844] Serialized 14 byte log entry
0430 07:59:09.777640 (+142729us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:09.777695 (+ 55us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:09.777748 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:09.777828 (+ 80us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:09.777972 (+ 144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":540979,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":11428,"thread_start_us":1002810,"threads_started":2}
I20260430 07:59:09.786962 9858 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:09.749755 11545 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.788369 10007 rpcz_store.cc:276] 0430 07:59:07.664627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:07.706631 (+ 42004us) service_pool.cc:225] Handling call
0430 07:59:07.706949 (+ 318us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:07.819659 (+112710us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:07.819709 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:07.838101 (+ 18392us) log.cc:844] Serialized 26 byte log entry
0430 07:59:08.181790 (+343689us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:08.181847 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:08.181952 (+ 105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:09.682385 (+1500433us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:09.723479 (+ 41094us) raft_consensus.cc:1685] finished
0430 07:59:09.723503 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:09.764820 (+ 41317us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6577,"raft.queue_time_us":76327,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":11809,"spinlock_wait_cycles":25810688,"thread_start_us":441516,"threads_started":2,"wal-append.queue_time_us":563560}
I20260430 07:59:09.758129 12087 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:09.754010 12000 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:09.794514 12054 consensus_meta.cc:287] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.519s user 0.008s sys 0.004s
I20260430 07:59:09.697692 11918 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.797389 9938 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:09.799489 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 223094us
I20260430 07:59:09.805112 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:09.749511 12076 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:09.808634 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.789239 11789 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:09.830363 9936 raft_consensus.cc:2393] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:09.831703 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:09.834681 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 82515us
W20260430 07:59:09.835706 9828 leader_election.cc:336] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.014s (SENT)
I20260430 07:59:09.836331 9938 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:09.836555 9828 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:09.837765 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.824819 9934 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.824955 9858 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.843003 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.843745 9938 raft_consensus.cc:2368] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4.
I20260430 07:59:09.844156 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 954) took 1133 ms. Trace:
I20260430 07:59:09.844430 9938 rpcz_store.cc:276] 0430 07:59:08.710688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.842586 (+1131898us) service_pool.cc:225] Handling call
0430 07:59:09.844100 (+ 1514us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:09.845253 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.772588 12012 consensus_queue.cc:1059] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260430 07:59:09.846200 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 955) took 1135 ms. Trace:
I20260430 07:59:09.806471 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:09.852195 9938 rpcz_store.cc:276] 0430 07:59:08.711062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.845016 (+1133954us) service_pool.cc:225] Handling call
0430 07:59:09.846128 (+ 1112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:09.855356 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.856003 9858 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:09.856230 9938 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:09.811571 11724 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.770853 11802 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:09.864005 9828 leader_election.cc:336] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.449s (SENT)
I20260430 07:59:09.864835 9828 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:09.867516 9828 leader_election.cc:336] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.596s (SENT)
I20260430 07:59:09.868140 9828 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:09.795794 11952 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.869357 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 956) took 1127 ms. Trace:
I20260430 07:59:09.870065 9938 rpcz_store.cc:276] 0430 07:59:08.743857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.855102 (+1111245us) service_pool.cc:225] Handling call
0430 07:59:09.869294 (+ 14192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:09.870985 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.871506 9938 raft_consensus.cc:3060] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:09.872532 9858 raft_consensus.cc:1275] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:09.846515 12076 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.827700 12054 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.862816 10562 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:09.872552 9904 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:09.842391 12004 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.724110 12060 consensus_queue.cc:237] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:09.862222 11738 consensus_meta.cc:287] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.752s user 0.005s sys 0.006s
I20260430 07:59:09.828233 12064 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:09.889990 9828 leader_election.cc:336] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 3.001s (SENT)
I20260430 07:59:09.903565 12028 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:09.903806 9828 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:09.906565 12009 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.907356 12009 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:09.907466 9828 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:09.870288 12079 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:09.846871 11789 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.890501 12051 raft_consensus.cc:2749] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:09.809403 11545 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.904969 11802 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.897526 9932 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.930971 9938 raft_consensus.cc:2468] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:09.863735 12077 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.913116 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.914964 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:09.935338 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:09.904713 11849 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:09.863152 11841 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:09.829550 12000 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.920703 9937 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:09.908249 12009 raft_consensus.cc:697] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:09.901404 11738 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:09.907287 11952 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.924249 12080 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:09.877818 11724 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.952509 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:09.955427 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 48013.1us
I20260430 07:59:09.903578 12076 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:09.961648 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.931588 12065 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:09.972370 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:09.971616 9938 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:09.908730 12004 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.938431 12096 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:09.974735 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:09.974109 9828 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:09.977911 12076 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:09.982272 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.904309 12028 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:09.979837 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:09.982501 12076 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:09.985589 9858 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 5
I20260430 07:59:09.989140 9938 raft_consensus.cc:2393] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:09.976773 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:09.986480 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:09.971953 11974 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.993489 11952 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260430 07:59:10.000029 12101 log.cc:927] Time spent T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260430 07:59:10.000396 12009 consensus_queue.cc:237] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.001130 12030 raft_consensus.cc:3055] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:10.001971 12030 raft_consensus.cc:740] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:10.003180 12000 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:10.003039 12030 consensus_queue.cc:260] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:10.004520 9828 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:10.006139 12100 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260430 07:59:10.006709 11974 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:09.984545 12065 raft_consensus.cc:697] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:10.008491 12065 consensus_queue.cc:237] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:10.009938 12054 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:10.006016 9936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 959) took 1176 ms (client timeout 1550 ms). Trace:
W20260430 07:59:10.011960 9936 rpcz_store.cc:269] 0430 07:59:08.829537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.012871 (+183334us) service_pool.cc:225] Handling call
0430 07:59:10.005797 (+992926us) spinlock_profiling.cc:243] Waited 762 ms on lock 0x7b5400348854. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:10.005954 (+ 157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1752026496}
I20260430 07:59:09.988479 11841 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:10.013096 12028 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:09.996850 12050 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.014271 12015 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:10.013163 12004 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.006011 9828 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:10.013262 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.021847 9828 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:10.022841 9936 raft_consensus.cc:2393] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 3: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:10.026144 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.026811 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.026856 9860 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:59:09.986575 12096 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.029922 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.030536 9861 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:10.031728 12096 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.031601 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:09.990448 12076 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:10.032788 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 958) took 1219 ms. Trace:
I20260430 07:59:10.033224 9936 rpcz_store.cc:276] 0430 07:59:08.813017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.026714 (+1213697us) service_pool.cc:225] Handling call
0430 07:59:10.032713 (+ 5999us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:10.033336 11794 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:10.034878 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.037866 9937 raft_consensus.cc:2410] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:10.040865 12110 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:10.041375 12000 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:10.046131 9828 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:10.044263 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 210125us
I20260430 07:59:10.039237 10534 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:10.050823 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.055330 12050 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:10.055693 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:10.055984 12050 raft_consensus.cc:697] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:10.053010 11952 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:10.057261 11952 raft_consensus.cc:697] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:10.057924 12012 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:10.057013 12050 consensus_queue.cc:237] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.058357 12012 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:10.058779 11952 consensus_queue.cc:237] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.059736 11545 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:10.060777 9861 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:10.067682 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.068397 9861 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:10.066702 9904 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:10.070698 9934 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:10.071408 10534 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:10.071710 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.054005 11850 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.075583 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:10.082405 12043 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:10.083422 12043 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:10.063908 12110 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:10.084090 12043 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:10.088248 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:10.046571 11918 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:10.058734 12012 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:10.083699 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:10.105729 9854 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260430 07:59:10.121033 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:10.297565 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 185090us
I20260430 07:59:10.124675 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.317081 9903 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:10.194841 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 949) took 1507 ms. Trace:
I20260430 07:59:10.318238 9937 rpcz_store.cc:276] 0430 07:59:08.687267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.961082 (+1273815us) service_pool.cc:225] Handling call
0430 07:59:10.194575 (+233493us) spinlock_profiling.cc:243] Waited 64.9 ms on lock 0x7b54002a0654. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:10.194768 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149288704}
I20260430 07:59:10.320967 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.298606 9858 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:10.332666 12028 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260430 07:59:10.333668 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 234) took 1136 ms. Trace:
I20260430 07:59:10.192119 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:59:10.101205 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.299063 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.336452 9855 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:59:10.338459 9828 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.887s (SENT) [suppressed 50 similar messages]
I20260430 07:59:10.069972 12000 raft_consensus.cc:697] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:10.313084 9932 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
W20260430 07:59:10.342182 9828 leader_election.cc:336] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.050s (SENT)
I20260430 07:59:10.342959 9828 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:10.326931 9903 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:10.328570 11769 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:10.086221 12110 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:10.331665 12103 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:10.130661 12002 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:10.323792 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:10.241367 11742 consensus_meta.cc:287] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.522s user 0.013s sys 0.000s
W20260430 07:59:10.243696 11623 consensus_meta.cc:287] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.527s user 0.010s sys 0.000s
W20260430 07:59:10.127640 12036 log.cc:927] Time spent T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 1.567s user 0.000s sys 0.010s
I20260430 07:59:10.327674 9861 raft_consensus.cc:2393] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 4: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:10.346607 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.367772 9856 raft_consensus.cc:2410] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260430 07:59:10.259655 11724 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.096788 11850 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:10.235445 12065 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:10.348742 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:10.126940 11802 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:10.350030 9828 leader_election.cc:336] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.124s (SENT)
I20260430 07:59:10.128746 12097 consensus_queue.cc:1059] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.288s
I20260430 07:59:10.352666 9863 raft_consensus.cc:1240] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:10.162520 11789 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.334097 9937 rpcz_store.cc:276] 0430 07:59:09.197642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.320647 (+1123005us) service_pool.cc:225] Handling call
0430 07:59:10.333605 (+ 12958us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:10.337880 12033 consensus_meta.cc:287] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.619s user 0.011s sys 0.002s
I20260430 07:59:10.131222 11738 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.349148 12060 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260430 07:59:10.388470 12114 log.cc:927] Time spent T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.203s user 0.001s sys 0.000s
I20260430 07:59:10.353735 10561 raft_consensus.cc:2368] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
W20260430 07:59:10.389436 10561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54770 (request call id 340) took 1758 ms (client timeout 1569 ms). Trace:
W20260430 07:59:10.389902 10561 rpcz_store.cc:269] 0430 07:59:08.631407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:08.631612 (+ 205us) service_pool.cc:225] Handling call
0430 07:59:10.083442 (+1451830us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b58001f1798. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:10.389350 (+305908us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31912192}
W20260430 07:59:10.391041 12113 log.cc:927] Time spent T f8946312605d455cbfe6546ad6873ec1 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
W20260430 07:59:10.391989 12022 consensus_meta.cc:287] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.581s user 0.012s sys 0.001s
I20260430 07:59:10.392045 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.393116 9937 raft_consensus.cc:2393] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:10.394122 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.392694 12022 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:10.357300 11742 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.246942 12035 consensus_queue.cc:1059] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.286s
I20260430 07:59:10.384303 9828 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:10.396097 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 46488.6us
I20260430 07:59:10.396175 12060 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:59:10.396763 9828 leader_election.cc:336] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.207s (SENT)
I20260430 07:59:10.397224 9828 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:10.398041 12060 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:59:10.354064 12002 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:10.351419 9903 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:10.399000 12002 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:10.387337 12065 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.400038 11742 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.394690 9937 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:10.401635 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.402245 9937 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:10.402717 12022 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:10.403072 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 973) took 1309 ms. Trace:
I20260430 07:59:10.403455 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.238432 12087 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.359817 11765 raft_consensus.cc:2749] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:10.403964 12065 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.370344 11769 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:10.388470 11738 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:10.401993 12040 log.cc:927] Time spent T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 1.875s user 0.004s sys 0.001s
W20260430 07:59:10.398052 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:10.409564 10534 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:10.403504 9938 rpcz_store.cc:276] 0430 07:59:09.094088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.981940 (+887852us) service_pool.cc:225] Handling call
0430 07:59:10.402873 (+420933us) spinlock_profiling.cc:243] Waited 5.74 ms on lock 0x7b54000b4fd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:10.403013 (+ 140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13211520}
I20260430 07:59:10.404507 9937 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:10.389739 12000 consensus_queue.cc:237] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.368057 9904 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:10.405673 9934 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
W20260430 07:59:10.398522 9828 leader_election.cc:336] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.209s (SENT)
W20260430 07:59:10.385870 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 62347.8us
I20260430 07:59:10.425487 9828 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:10.425827 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:10.426728 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:10.411339 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 60010.2us
I20260430 07:59:10.426689 9938 raft_consensus.cc:2410] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260430 07:59:10.428709 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:10.429447 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:10.412374 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.416347 9854 raft_consensus.cc:1240] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:10.409564 11769 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:10.431421 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:10.432508 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 64554.6us
I20260430 07:59:10.432772 9936 raft_consensus.cc:2368] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6.
I20260430 07:59:10.426927 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1367) took 2541 ms. Trace:
I20260430 07:59:10.433518 9937 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:10.436146 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 947) took 1749 ms. Trace:
I20260430 07:59:10.435788 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.437026 9937 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:10.436775 9938 rpcz_store.cc:276] 0430 07:59:08.686613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.425562 (+1738949us) service_pool.cc:225] Handling call
0430 07:59:10.435905 (+ 10343us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:10.438133 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:10.439673 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:10.438539 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.441344 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 948) took 1754 ms. Trace:
I20260430 07:59:10.369210 12076 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:10.433822 9933 rpcz_store.cc:276] 0430 07:59:07.884462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:08.094904 (+210442us) service_pool.cc:225] Handling call
0430 07:59:08.095334 (+ 430us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:08.151187 (+ 55853us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:08.151220 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:08.152552 (+ 1332us) log.cc:844] Serialized 52 byte log entry
0430 07:59:08.237546 (+ 84994us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:08.527342 (+289796us) log.cc:844] Serialized 14 byte log entry
0430 07:59:08.557866 (+ 30524us) log.cc:844] Serialized 14 byte log entry
0430 07:59:08.560296 (+ 2430us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:08.560373 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:09.632320 (+1071947us) spinlock_profiling.cc:243] Waited 6.64 ms on lock 0x7b18003db2e0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661b3be6 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56e1bb1a1 00007fb56e1b2237 00007fb56e1b20ca 00007fb56e2a992f
0430 07:59:09.635767 (+ 3447us) spinlock_profiling.cc:243] Waited 3.43 ms on lock 0x7b48000d26d8. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56e1bb1a1 00007fb56e1b2237 00007fb56e1b20ca 00007fb56e2a992f 00007fb56e2a95f7 00007fb56e2bdc1c 00007fb56e2bdbc8 00007fb56e2bdb48 00007fb56e2bdb03 00007fb56e2bcdac 00007fb56d1e5b36
0430 07:59:10.127312 (+491545us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b2400066698. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb566037e94 00007fb566035f9b 00007fb5660c4d39 00007fb5660c49c5 00007fb5660c400d 00007fb568e064ae 00007fb568e1dc2c 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1
0430 07:59:10.409731 (+282419us) raft_consensus.cc:1685] finished
0430 07:59:10.409755 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:10.409931 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":145431,"prepare.queue_time_us":833023,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":3171,"raft.queue_time_us":89608,"raft.run_cpu_time_us":3427,"raft.run_wall_time_us":1088856,"spinlock_wait_cycles":54037120,"thread_start_us":230964,"threads_started":2,"wal-append.queue_time_us":273568}
W20260430 07:59:10.499367 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 66196.1us
I20260430 07:59:10.500159 9938 rpcz_store.cc:276] 0430 07:59:08.686931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.437972 (+1751041us) service_pool.cc:225] Handling call
0430 07:59:10.441243 (+ 3271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:10.503309 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:10.505309 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:10.505961 9938 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:10.499099 9937 raft_consensus.cc:2410] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260430 07:59:10.439673 9828 leader_election.cc:336] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.240s (SENT)
I20260430 07:59:10.370242 11623 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:10.428059 12110 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:10.514796 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 75131.5us
I20260430 07:59:10.432261 11952 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:10.516983 9828 leader_election.cc:336] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.459s (SENT)
I20260430 07:59:10.521144 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1371) took 2616 ms. Trace:
I20260430 07:59:10.518296 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.521001 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.522328 9933 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:10.521727 9931 rpcz_store.cc:276] 0430 07:59:07.905153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:08.144864 (+239711us) service_pool.cc:225] Handling call
0430 07:59:08.145191 (+ 327us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:08.151294 (+ 6103us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:08.151323 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:08.171075 (+ 19752us) log.cc:844] Serialized 52 byte log entry
0430 07:59:08.337563 (+166488us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:08.809576 (+472013us) log.cc:844] Serialized 14 byte log entry
0430 07:59:08.842551 (+ 32975us) log.cc:844] Serialized 14 byte log entry
0430 07:59:08.842802 (+ 251us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:08.842887 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:09.313035 (+470148us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b180048bda0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661e5a33 00007fb5661e7e01 00007fb5661e7db9 00007fb5661e7d49 00007fb5661e7d11 00007fb5661e700d 0000000000379254 0000000000379099 00007fb5661c8fba
0430 07:59:09.508349 (+195314us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b180048bda0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661e5a33 00007fb5661e7e01 00007fb5661e7db9 00007fb5661e7d49 00007fb5661e7d11 00007fb5661e700d 0000000000379254 0000000000379099 00007fb5661c8fba
0430 07:59:09.624560 (+116211us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b48000d26d8. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56e1bb1a1 00007fb56e1b2237 00007fb56e1b20ca 00007fb56e2a992f 00007fb56e2a95f7 00007fb56e2bdc1c 00007fb56e2bdbc8 00007fb56e2bdb48 00007fb56e2bdb03 00007fb56e2bcdac 00007fb56d1e5b36
0430 07:59:10.401229 (+776669us) spinlock_profiling.cc:243] Waited 4.78 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:10.520923 (+119694us) raft_consensus.cc:1685] finished
0430 07:59:10.520947 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:10.521085 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":160235,"prepare.queue_time_us":1022418,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":95376,"raft.run_cpu_time_us":3025,"raft.run_wall_time_us":1484323,"spinlock_wait_cycles":96925312,"thread_start_us":460953,"threads_started":2,"wal-append.queue_time_us":218039}
I20260430 07:59:10.524324 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:10.507797 12036 log.cc:927] Time spent T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20260430 07:59:10.525738 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:10.358070 12103 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.527755 9938 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:10.528205 12076 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:10.528774 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:10.529026 12103 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.530364 12087 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.530471 11769 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:10.531680 12002 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.532223 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1459) took 2917 ms. Trace:
I20260430 07:59:10.532645 9862 rpcz_store.cc:276] 0430 07:59:07.615283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:07.628431 (+ 13148us) service_pool.cc:225] Handling call
0430 07:59:07.628760 (+ 329us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:07.709264 (+ 80504us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:07.709354 (+ 90us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:07.709423 (+ 69us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:07.709459 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:10.531996 (+2822537us) spinlock_profiling.cc:243] Waited 80.3 ms on lock 0x7b540010c554. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:10.532044 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:10.532167 (+ 123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":184606080}
I20260430 07:59:10.534121 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:10.405920 11802 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.497483 11724 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:10.546373 11724 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:10.546800 11724 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:10.530558 9938 raft_consensus.cc:1275] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260430 07:59:10.536410 9828 leader_election.cc:336] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 3.259s (SENT)
I20260430 07:59:10.549763 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1373) took 2629 ms. Trace:
I20260430 07:59:10.550123 9828 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:10.551507 9828 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:10.552634 11802 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:10.555366 10564 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54770 (request call id 338) took 2811 ms (client timeout 1579 ms). Trace:
W20260430 07:59:10.555881 10564 rpcz_store.cc:269] 0430 07:59:07.743611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:07.746074 (+ 2463us) service_pool.cc:225] Handling call
0430 07:59:10.555168 (+2809094us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b540063c4d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:10.555287 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4064258816}
I20260430 07:59:10.557938 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1501) took 1178 ms. Trace:
I20260430 07:59:10.558183 11841 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:10.558377 9859 rpcz_store.cc:276] 0430 07:59:09.379848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.614013 (+234165us) service_pool.cc:225] Handling call
0430 07:59:09.673962 (+ 59949us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:09.694913 (+ 20951us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:10.557707 (+862794us) spinlock_profiling.cc:243] Waited 35.4 ms on lock 0x7b540029cf50. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:10.557867 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":81379200}
I20260430 07:59:10.558894 11841 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:10.517112 12097 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:10.540820 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:10.541743 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:10.569168 9863 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:10.552578 9828 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:10.569922 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 140503us
I20260430 07:59:10.560230 9937 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:10.550143 9938 rpcz_store.cc:276] 0430 07:59:07.920765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.529942 (+2609177us) service_pool.cc:225] Handling call
0430 07:59:10.530244 (+ 302us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:10.549469 (+ 19225us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:10.549705 (+ 236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:10.539417 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.557878 12012 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:10.545855 10561 raft_consensus.cc:1240] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:10.577668 9938 raft_consensus.cc:1240] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:10.579953 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.581298 9862 raft_consensus.cc:2410] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260430 07:59:10.574772 9828 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:10.588147 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:10.605913 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:10.609249 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1375) took 2657 ms. Trace:
I20260430 07:59:10.609858 9860 raft_consensus.cc:2410] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260430 07:59:10.609771 9938 rpcz_store.cc:276] 0430 07:59:07.937005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.576869 (+2639864us) service_pool.cc:225] Handling call
0430 07:59:10.577461 (+ 592us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:10.578277 (+ 816us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:10.578558 (+ 281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:10.611620 9938 raft_consensus.cc:1240] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:10.612856 10567 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:10.614570 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:10.617012 9828 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:10.619081 9828 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:10.508158 11765 consensus_queue.cc:1059] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20260430 07:59:10.567839 11995 raft_consensus.cc:3055] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:10.643884 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:10.644639 10565 raft_consensus.cc:2376] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3. Re-sending same reply.
I20260430 07:59:10.606904 12065 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:10.565136 12122 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:10.639948 11995 raft_consensus.cc:740] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:10.676961 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:10.575624 12097 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.676332 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:10.683832 10567 raft_consensus.cc:2368] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:10.641669 11841 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:10.641656 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1376) took 2702 ms. Trace:
I20260430 07:59:10.697319 9938 rpcz_store.cc:276] 0430 07:59:07.937424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.610993 (+2673569us) service_pool.cc:225] Handling call
0430 07:59:10.611397 (+ 404us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:10.612058 (+ 661us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:10.639645 (+ 27587us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:10.636222 12051 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:10.716403 9933 raft_consensus.cc:2393] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:10.674801 12033 raft_consensus.cc:3055] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
W20260430 07:59:10.719255 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:10.690431 10562 raft_consensus.cc:2368] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:10.708240 11623 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.676330 12121 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:10.683386 12002 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.744140 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:10.744964 11995 consensus_queue.cc:260] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.751262 11995 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:10.753986 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:10.754778 10567 raft_consensus.cc:2368] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:10.756881 9860 raft_consensus.cc:1275] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260430 07:59:10.742116 12124 log.cc:927] Time spent T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260430 07:59:10.721663 11841 raft_consensus.cc:697] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:10.730755 12033 raft_consensus.cc:740] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:10.761353 12121 raft_consensus.cc:697] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:10.764480 11802 consensus_queue.cc:1048] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
W20260430 07:59:10.786182 12043 consensus_meta.cc:287] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.701s user 0.014s sys 0.003s
I20260430 07:59:10.798000 12043 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:10.801075 12121 consensus_queue.cc:237] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:10.804085 10561 raft_consensus.cc:1240] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:10.795241 12012 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:10.795732 12033 consensus_queue.cc:260] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.797475 9857 raft_consensus.cc:1240] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:59:10.788762 11841 consensus_queue.cc:237] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.854771 9863 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:10.861857 12097 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:10.869347 9904 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:10.871031 12002 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:10.871968 12002 raft_consensus.cc:697] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:10.873194 12002 consensus_queue.cc:237] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:10.879230 11952 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:10.837598 12043 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:10.852116 9828 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:10.825235 11724 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:10.903080 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 281804us
I20260430 07:59:10.903630 9931 raft_consensus.cc:2368] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:10.905964 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:10.909845 12000 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:10.945178 12000 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:10.945879 12000 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:10.947904 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:10.904474 9828 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:10.949805 11841 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:59:10.963966 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:10.950069 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 45656.3us
I20260430 07:59:10.969929 9828 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:10.971127 11841 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:10.971560 9828 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:10.972471 12060 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:10.972993 12060 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:10.973542 12060 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:10.979763 11841 raft_consensus.cc:2749] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:10.915721 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 46492.4us
I20260430 07:59:10.921962 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 225) took 2981 ms. Trace:
I20260430 07:59:10.972530 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:11.010389 9828 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:10.991084 9938 rpcz_store.cc:276] 0430 07:59:07.940547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.698839 (+2758292us) service_pool.cc:225] Handling call
0430 07:59:10.699336 (+ 497us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:10.843010 (+143674us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:10.843057 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:10.844112 (+ 1055us) log.cc:844] Serialized 52 byte log entry
0430 07:59:10.921408 (+ 77296us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:10.921565 (+ 157us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:10.921652 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:10.921712 (+ 60us) raft_consensus.cc:1685] finished
0430 07:59:10.921725 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:10.921895 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":143046,"raft.queue_time_us":162826,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":76281,"threads_started":1,"wal-append.queue_time_us":52010}
I20260430 07:59:11.013206 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:11.014842 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:11.017170 9862 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:10.904711 11724 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:11.019415 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:10.940663 11619 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:11.029892 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:11.030786 9857 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 5
I20260430 07:59:11.020891 9904 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:11.018980 12050 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:11.015481 11841 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:11.040756 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:11.041739 10567 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:11.020181 11802 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.015947 12097 consensus_queue.cc:1059] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260430 07:59:11.048628 11769 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:11.045410 12051 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:11.040748 12050 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:11.048475 11841 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:11.054325 11802 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.066781 12078 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:11.069631 11841 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:11.075498 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:11.056854 12004 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.073974 12051 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.077996 12004 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.080247 12004 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:11.063417 11769 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:11.064301 12050 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:11.088492 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:11.088508 11769 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:11.091702 12060 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.069324 11802 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:11.092847 9929 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1352) took 3270 ms (client timeout 2108 ms). Trace:
W20260430 07:59:11.093410 9929 rpcz_store.cc:269] 0430 07:59:07.822613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:07.822764 (+ 151us) service_pool.cc:225] Handling call
0430 07:59:11.092650 (+3269886us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400323cd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:11.092766 (+ 116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2931345152}
I20260430 07:59:11.094785 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:11.079787 12051 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:11.097783 9936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 950) took 2410 ms (client timeout 2596 ms). Trace:
W20260430 07:59:11.098217 9936 rpcz_store.cc:269] 0430 07:59:08.687612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.034456 (+1346844us) service_pool.cc:225] Handling call
0430 07:59:11.097592 (+1063136us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b5400347e54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:11.097723 (+ 131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":912647936}
I20260430 07:59:11.099236 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:11.099938 9936 raft_consensus.cc:2368] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6.
I20260430 07:59:11.100660 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:11.103641 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:11.106215 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 226) took 3163 ms. Trace:
I20260430 07:59:11.106593 12013 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:11.101159 9936 raft_consensus.cc:2376] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6. Re-sending same reply.
I20260430 07:59:11.107841 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:11.108590 9936 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:59:11.106670 9929 rpcz_store.cc:276] 0430 07:59:07.942790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.105343 (+3162553us) service_pool.cc:225] Handling call
0430 07:59:11.105637 (+ 294us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:11.105810 (+ 173us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:11.105834 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:11.105875 (+ 41us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:11.105903 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.105951 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:11.106049 (+ 98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:11.111451 12060 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:11.111897 9929 pending_rounds.cc:85] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 2
I20260430 07:59:11.112241 9929 pending_rounds.cc:107] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260430 07:59:11.112463 9929 raft_consensus.cc:2889] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260430 07:59:11.113047 9934 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58726 (request call id 236) took 1852 ms (client timeout 2213 ms). Trace:
W20260430 07:59:11.113476 9934 rpcz_store.cc:269] 0430 07:59:09.260707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.076697 (+815990us) service_pool.cc:225] Handling call
0430 07:59:11.112888 (+1036191us) spinlock_profiling.cc:243] Waited 68.9 ms on lock 0x7b540003be54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:11.112992 (+ 104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":158558080}
I20260430 07:59:11.117879 9857 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
W20260430 07:59:11.108776 12038 log.cc:927] Time spent T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 2.548s user 0.000s sys 0.012s
I20260430 07:59:11.120057 9936 raft_consensus.cc:1240] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260430 07:59:11.118050 10567 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:11.118773 9934 raft_consensus.cc:2955] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:11.124078 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1380) took 2956 ms. Trace:
I20260430 07:59:11.124607 9936 rpcz_store.cc:276] 0430 07:59:08.167513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.115538 (+2948025us) service_pool.cc:225] Handling call
0430 07:59:11.115953 (+ 415us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:11.123143 (+ 7190us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.124018 (+ 875us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:11.125705 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:11.126430 10564 raft_consensus.cc:1240] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:11.128445 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1370) took 3223 ms. Trace:
I20260430 07:59:11.128844 9935 rpcz_store.cc:276] 0430 07:59:07.904835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:08.138055 (+233220us) service_pool.cc:225] Handling call
0430 07:59:08.138359 (+ 304us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:08.161095 (+ 22736us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:08.161126 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:08.179175 (+ 18049us) log.cc:844] Serialized 26 byte log entry
0430 07:59:08.333753 (+154578us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:08.579808 (+246055us) log.cc:844] Serialized 14 byte log entry
0430 07:59:08.584204 (+ 4396us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:08.584346 (+ 142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:11.104025 (+2519679us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b18002e77a0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661e5b62 00007fb5661e7e01 00007fb5661e7db9 00007fb5661e7d49 00007fb5661e7d11 00007fb5661e700d 0000000000379254 0000000000379099 00007fb5661c8fba
0430 07:59:11.108179 (+ 4154us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:11.128223 (+ 20044us) raft_consensus.cc:1685] finished
0430 07:59:11.128247 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:11.128379 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28985,"prepare.queue_time_us":448274,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":20658,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":19590,"spinlock_wait_cycles":30898816,"thread_start_us":347019,"threads_started":2,"wal-append.queue_time_us":285210}
I20260430 07:59:11.131915 12030 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:11.137094 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:11.143371 9828 leader_election.cc:400] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6.
I20260430 07:59:11.145443 11952 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:11.145974 11748 consensus_queue.cc:1059] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260430 07:59:11.149199 11748 consensus_queue.cc:1243] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer 27f97a330f374849be58a168318c1ec6 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:59:11.145256 12051 consensus_queue.cc:1059] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260430 07:59:11.144158 9904 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:11.081658 12000 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.149089 11952 raft_consensus.cc:697] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:11.153985 12133 raft_consensus.cc:3055] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:11.155238 12133 raft_consensus.cc:740] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:11.155376 11952 consensus_queue.cc:237] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:11.156337 12133 consensus_queue.cc:260] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:11.157641 12133 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:11.159184 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 984) took 1247 ms. Trace:
I20260430 07:59:11.159704 9931 rpcz_store.cc:276] 0430 07:59:09.911511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.524068 (+612557us) service_pool.cc:225] Handling call
0430 07:59:11.158951 (+634883us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b5400347454. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:11.159122 (+ 171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":870208256}
I20260430 07:59:11.160871 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:11.162047 12000 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:11.108904 12050 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.165073 9931 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:11.239455 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:11.244396 9828 leader_election.cc:400] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:11.244932 9828 leader_election.cc:403] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260430 07:59:11.173161 11748 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:59:11.326555 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 82653.3us
I20260430 07:59:11.327589 9828 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:11.238839 12060 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:11.276118 12051 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.278630 11724 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.386315 9903 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:11.390054 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:11.311873 12103 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.385295 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:11.408802 9931 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:11.376346 12078 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
I20260430 07:59:11.377043 12011 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.383526 12000 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:11.414412 9930 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1351) took 3597 ms (client timeout 1724 ms). Trace:
W20260430 07:59:11.417467 9977 connection.cc:254] RPC call timeout handler was delayed by -3.5340074900000218s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3309s for a response
I20260430 07:59:11.387389 12076 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:11.420225 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:11.417937 9930 rpcz_store.cc:269] 0430 07:59:07.816958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:07.819240 (+ 2282us) service_pool.cc:225] Handling call
0430 07:59:11.414045 (+3594805us) spinlock_profiling.cc:243] Waited 82 ms on lock 0x7b540036bfd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:11.414342 (+ 297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188533120}
I20260430 07:59:11.446278 10003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 203) took 2571 ms. Trace:
I20260430 07:59:11.446573 10567 raft_consensus.cc:2368] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5.
I20260430 07:59:11.395743 12051 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:11.450944 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 231) took 2519 ms. Trace:
I20260430 07:59:11.468112 9930 rpcz_store.cc:276] 0430 07:59:08.931297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.446183 (+2514886us) service_pool.cc:225] Handling call
0430 07:59:11.450414 (+ 4231us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:11.450604 (+ 190us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:11.450628 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:11.450669 (+ 41us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:11.450700 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.450751 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:11.450886 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:11.446750 10003 rpcz_store.cc:276] 0430 07:59:08.836419 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:08.866266 (+ 29847us) service_pool.cc:225] Handling call
0430 07:59:08.912452 (+ 46186us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:08.912684 (+ 232us) raft_consensus.cc:1534] Early marking committed up to index 8
0430 07:59:09.257692 (+345008us) log.cc:844] Serialized 14 byte log entry
0430 07:59:09.582927 (+325235us) log.cc:844] Serialized 14 byte log entry
0430 07:59:11.407343 (+1824416us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b18002aacc0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb566159e1a 00007fb56f7f25c9 00007fb5660181f0 00007fb5661e2b24 00007fb5661e2a95 00007fb56927b440 00007fb56928daf6 00007fb56929e667 00007fb56929e5e8
0430 07:59:11.407545 (+ 202us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:11.407596 (+ 51us) raft_consensus.cc:1645] Marking committed up to 8
0430 07:59:11.407640 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.407719 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:11.407883 (+ 164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1828817,"prepare.queue_time_us":752186,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":6547,"raft.queue_time_us":1845544,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":8830,"spinlock_wait_cycles":3282176,"thread_start_us":624820,"threads_started":2,"wal-append.queue_time_us":309773,"wal-append.run_cpu_time_us":1837,"wal-append.run_wall_time_us":1047591}
W20260430 07:59:11.484625 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:11.497404 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 232) took 2508 ms. Trace:
I20260430 07:59:11.500142 9930 rpcz_store.cc:276] 0430 07:59:08.989196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.469294 (+2480098us) service_pool.cc:225] Handling call
0430 07:59:11.496780 (+ 27486us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:11.496988 (+ 208us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:11.497019 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:11.497072 (+ 53us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:11.497110 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.497168 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:11.497341 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:11.503568 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:11.416126 12065 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:11.420296 12000 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:11.399250 12060 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.398844 11724 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.518662 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:11.408174 12103 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:11.502663 12118 log.cc:927] Time spent T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260430 07:59:11.448345 11802 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.519379 9860 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:59:11.496690 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:11.524314 9931 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:11.420001 12011 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.538452 12011 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:11.452273 12076 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.519757 12000 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:11.545679 9931 raft_consensus.cc:1275] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260430 07:59:11.547472 9935 raft_consensus.cc:1275] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260430 07:59:11.556898 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 71187.4us
W20260430 07:59:11.552656 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 223426us
I20260430 07:59:11.516152 12078 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:11.561348 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:11.562453 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:11.525607 11748 raft_consensus.cc:697] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:11.552274 11501 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:11.577507 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:11.577689 11501 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:11.564543 11850 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:11.578742 11850 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.528252 11802 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:11.652981 11841 consensus_meta.cc:287] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.583s user 0.007s sys 0.004s
I20260430 07:59:11.653962 11802 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:11.581106 9977 connection.cc:254] RPC call timeout handler was delayed by -2.9340234779999719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16190000000000002s for a response
I20260430 07:59:11.581243 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:11.655661 9977 connection.cc:254] RPC call timeout handler was delayed by -2.9106234779999909s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1645s for a response
I20260430 07:59:11.654342 11841 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:11.647899 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 90377.2us
I20260430 07:59:11.580320 11501 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:11.574717 11336 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.658165 11841 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:11.658066 11336 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:11.656201 9857 raft_consensus.cc:2393] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:11.660238 11336 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:11.581326 11986 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:11.661442 11986 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:11.605640 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:11.575016 12078 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:11.611956 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:11.613246 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 227) took 3453 ms. Trace:
I20260430 07:59:11.522292 12065 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.582432 10564 raft_consensus.cc:2393] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 4: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
W20260430 07:59:11.679713 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:11.672281 9934 rpcz_store.cc:276] 0430 07:59:08.159443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.114240 (+2954797us) service_pool.cc:225] Handling call
0430 07:59:11.114557 (+ 317us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:11.114755 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:11.139915 (+ 25160us) log.cc:844] Serialized 14 byte log entry
0430 07:59:11.604790 (+464875us) log.cc:844] Serialized 14 byte log entry
0430 07:59:11.611211 (+ 6421us) log.cc:844] Serialized 14 byte log entry
0430 07:59:11.611442 (+ 231us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:11.611511 (+ 69us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:11.611579 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.611680 (+ 101us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:11.613153 (+ 1473us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":33564,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":100,"raft.queue_time_us":208667,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":148,"thread_start_us":416453,"threads_started":1}
I20260430 07:59:11.684582 9934 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:11.685550 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:11.656217 9977 connection.cc:254] RPC call timeout handler was delayed by -2.9097234779999894s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
W20260430 07:59:11.687476 9977 connection.cc:254] RPC call timeout handler was delayed by -2.8917234780000172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
W20260430 07:59:11.687850 9977 connection.cc:254] RPC call timeout handler was delayed by -2.8395234779999896s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1724s for a response
W20260430 07:59:11.688294 9977 connection.cc:254] RPC call timeout handler was delayed by -2.7486234780000132s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1825s for a response
I20260430 07:59:11.664357 11986 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:11.692699 10007 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:11.556340 12052 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.696846 10004 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:11.696846 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 242) took 1255 ms. Trace:
I20260430 07:59:11.698160 9934 rpcz_store.cc:276] 0430 07:59:10.431247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.683889 (+1252642us) service_pool.cc:225] Handling call
0430 07:59:11.684207 (+ 318us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:11.685154 (+ 947us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.686946 (+ 1792us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:11.699354 9934 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:11.700541 9934 raft_consensus.cc:3055] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:11.700973 9934 raft_consensus.cc:740] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:11.580591 11850 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:11.644672 11520 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.644991 12141 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:11.711668 10012 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260430 07:59:11.707635 9934 consensus_queue.cc:260] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.715803 9934 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:59:11.718638 9977 connection.cc:254] RPC call timeout handler was delayed by -2.6460234780000178s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19390000000000002s for a response
W20260430 07:59:11.719347 9977 connection.cc:254] RPC call timeout handler was delayed by -2.6262234779999858s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20260430 07:59:11.719832 9977 connection.cc:254] RPC call timeout handler was delayed by -2.5749234780000165s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
W20260430 07:59:11.720125 9977 connection.cc:254] RPC call timeout handler was delayed by -2.5632234779999976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
I20260430 07:59:11.672019 12011 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:11.707346 12031 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:11.730387 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:11.730932 12031 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:11.694700 11807 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:11.718540 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 61400.6us
I20260430 07:59:11.739671 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:11.723495 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:11.723611 11748 consensus_queue.cc:237] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:11.725539 12139 log.cc:927] Time spent T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.243s user 0.001s sys 0.000s
I20260430 07:59:11.706321 11336 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:11.729806 11724 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:11.729636 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:11.740542 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:11.756647 10004 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:11.741457 10010 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260430 07:59:11.720415 9977 connection.cc:254] RPC call timeout handler was delayed by -2.5272234779999962s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20710000000000003s for a response
I20260430 07:59:11.670490 12078 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:11.746156 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:11.705894 11501 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:11.747876 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:11.781605 10007 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 5
I20260430 07:59:11.750538 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:11.726615 12134 log.cc:927] Time spent T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.162s user 0.000s sys 0.000s
I20260430 07:59:11.723413 11626 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:11.711484 12050 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:11.779237 9977 connection.cc:254] RPC call timeout handler was delayed by -2.5218234779999875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2077s for a response
W20260430 07:59:11.790848 9977 connection.cc:254] RPC call timeout handler was delayed by -2.4939234779999992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21080000000000002s for a response
W20260430 07:59:11.791278 9977 connection.cc:254] RPC call timeout handler was delayed by -2.456123477999995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.215s for a response
I20260430 07:59:11.710464 12052 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.737764 11850 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:11.742734 12031 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:11.786173 9978 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:11.779121 10012 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:59:11.782944 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:11.799382 10534 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:59:11.791769 9977 connection.cc:254] RPC call timeout handler was delayed by -2.4543234779999921s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21520000000000003s for a response
I20260430 07:59:11.751551 11995 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:11.751502 11841 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:11.794222 9934 raft_consensus.cc:1275] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260430 07:59:11.751616 11952 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.003s sys 0.015s
I20260430 07:59:11.794386 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:11.794942 9860 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260430 07:59:11.797118 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:11.801298 9977 connection.cc:254] RPC call timeout handler was delayed by -2.1998533310000425s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
W20260430 07:59:11.810077 9977 connection.cc:254] RPC call timeout handler was delayed by -1.9845234779999714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2674s for a response
I20260430 07:59:11.806911 10006 raft_consensus.cc:2393] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:11.809607 11903 raft_consensus.cc:2804] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:11.812100 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1643) took 2236 ms. Trace:
W20260430 07:59:11.804126 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 417774us
I20260430 07:59:11.721711 11520 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.826045 9903 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:11.826839 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:11.801906 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:11.738931 12141 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:11.805940 9936 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s user 0.017s sys 0.003s
W20260430 07:59:11.810534 9977 connection.cc:254] RPC call timeout handler was delayed by -1.8722533310000244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21440000000000003s for a response
I20260430 07:59:11.812503 10005 rpcz_store.cc:276] 0430 07:59:09.575657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.575777 (+ 120us) service_pool.cc:225] Handling call
0430 07:59:09.591429 (+ 15652us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:09.640142 (+ 48713us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.811912 (+2171770us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b54000f82d0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:11.812045 (+ 133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2615552}
I20260430 07:59:11.812793 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:11.752596 11807 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.827759 10012 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:11.805403 11826 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:11.770957 11336 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:11.901729 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 75738.3us
I20260430 07:59:11.914932 9903 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:11.915555 11336 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:11.902264 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:11.916230 9903 leader_election.cc:400] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:11.916600 9903 leader_election.cc:403] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260430 07:59:11.916867 9862 raft_consensus.cc:2393] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 3: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:11.918251 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:11.918466 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:11.903460 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:11.919584 11769 consensus_meta.cc:287] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.828s user 0.008s sys 0.004s
I20260430 07:59:11.920138 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:11.920037 11769 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.903842 9934 raft_consensus.cc:3055] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:11.904362 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:11.904922 9977 connection.cc:254] RPC call timeout handler was delayed by -1.593923478000022s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3108s for a response
I20260430 07:59:11.809303 10003 raft_consensus.cc:2368] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:11.796550 12051 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:11.787220 11501 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.793956 11626 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:11.908325 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:11.910887 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:11.911067 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:11.903584 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:11.824690 11903 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:11.898520 11794 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.015s
I20260430 07:59:11.805708 11850 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:11.916926 11520 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:12.090787 12054 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:12.091814 12054 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.921310 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:11.977190 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.107848 9854 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:12.007121 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:11.973774 9856 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:11.985379 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1378) took 4043 ms. Trace:
I20260430 07:59:12.129717 9977 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:12.131459 9977 leader_election.cc:400] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5.
I20260430 07:59:12.131850 9977 leader_election.cc:403] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260430 07:59:12.133756 10012 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:12.162720 10006 raft_consensus.cc:2393] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:12.114619 9938 rpcz_store.cc:276] 0430 07:59:07.942342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.036134 (+3093792us) service_pool.cc:225] Handling call
0430 07:59:11.036652 (+ 518us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:11.053634 (+ 16982us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:11.053727 (+ 93us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:11.054494 (+ 767us) log.cc:844] Serialized 52 byte log entry
0430 07:59:11.362570 (+308076us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:11.437487 (+ 74917us) log.cc:844] Serialized 14 byte log entry
0430 07:59:11.482138 (+ 44651us) log.cc:844] Serialized 14 byte log entry
0430 07:59:11.795135 (+312997us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.795252 (+ 117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:11.985049 (+189797us) raft_consensus.cc:1685] finished
0430 07:59:11.985081 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:11.985304 (+ 223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64287,"prepare.queue_time_us":157169,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":119,"raft.queue_time_us":399919,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":3998,"thread_start_us":569480,"threads_started":2,"wal-append.queue_time_us":454920}
I20260430 07:59:12.165688 9938 raft_consensus.cc:1275] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:12.166563 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 997) took 1014 ms. Trace:
I20260430 07:59:12.166982 9938 rpcz_store.cc:276] 0430 07:59:11.151877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:12.164833 (+1012956us) service_pool.cc:225] Handling call
0430 07:59:12.165233 (+ 400us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:12.166211 (+ 978us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:12.166485 (+ 274us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:11.810511 11952 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s user 0.004s sys 0.019s
I20260430 07:59:12.198531 9934 raft_consensus.cc:740] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:12.169005 9938 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:12.209020 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:12.035475 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1381) took 3845 ms. Trace:
I20260430 07:59:12.176559 9860 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:59:11.814185 12060 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.207787 10004 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 7.
I20260430 07:59:12.199834 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.216250 10012 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:12.165786 10007 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:12.210767 9936 rpcz_store.cc:276] 0430 07:59:08.189713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.126518 (+2936805us) service_pool.cc:225] Handling call
0430 07:59:11.126882 (+ 364us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:11.127291 (+ 409us) raft_consensus.cc:1534] Early marking committed up to index 2
0430 07:59:11.146378 (+ 19087us) log.cc:844] Serialized 14 byte log entry
0430 07:59:11.898123 (+751745us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:11.899921 (+ 1798us) log.cc:844] Serialized 26 byte log entry
0430 07:59:11.900099 (+ 178us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:11.900727 (+ 628us) log.cc:844] Serialized 14 byte log entry
0430 07:59:11.901234 (+ 507us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.902256 (+ 1022us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:12.035133 (+132877us) raft_consensus.cc:1685] finished
0430 07:59:12.035165 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:12.035381 (+ 216us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26520,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":85,"raft.queue_time_us":568,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":90,"spinlock_wait_cycles":25210112,"thread_start_us":640171,"threads_started":2,"wal-append.queue_time_us":821874}
I20260430 07:59:12.213301 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:12.228297 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:12.244599 9904 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.843s user 0.014s sys 0.034s
W20260430 07:59:12.245141 9904 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.844s user 0.014s sys 0.034s
I20260430 07:59:12.253042 9858 raft_consensus.cc:1240] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:12.255045 9858 raft_consensus.cc:1240] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 2. Current term is 3. Ops: []
I20260430 07:59:12.256011 9828 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:12.368527 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 210274us
I20260430 07:59:12.373407 9859 raft_consensus.cc:1240] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
W20260430 07:59:11.997898 12030 consensus_meta.cc:287] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.865s user 0.012s sys 0.000s
W20260430 07:59:12.095467 12129 log.cc:927] Time spent T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20260430 07:59:12.378463 9855 raft_consensus.cc:1240] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:12.387578 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:12.388340 9855 raft_consensus.cc:2410] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260430 07:59:12.380244 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 124275us
I20260430 07:59:12.382364 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:12.382547 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.389596 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:11.995764 12135 log.cc:927] Time spent T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.420s user 0.000s sys 0.001s
I20260430 07:59:11.899711 12103 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.384325 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.434046 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1379) took 4290 ms. Trace:
I20260430 07:59:12.434295 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.435501 10004 raft_consensus.cc:2393] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:12.372480 11859 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5.
I20260430 07:59:11.970278 11826 raft_consensus.cc:697] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:12.389716 10007 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:12.437862 11826 consensus_queue.cc:237] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:11.813238 11841 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:12.443900 11841 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.417948 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:12.446846 10005 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:12.390291 9860 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:12.434621 9929 rpcz_store.cc:276] 0430 07:59:08.144046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.111407 (+2967361us) service_pool.cc:225] Handling call
0430 07:59:11.111730 (+ 323us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:11.112936 (+ 1206us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:11.112970 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:11.118477 (+ 5507us) log.cc:844] Serialized 26 byte log entry
0430 07:59:11.383281 (+264804us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:11.693912 (+310631us) log.cc:844] Serialized 14 byte log entry
0430 07:59:11.711111 (+ 17199us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:11.711331 (+ 220us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:12.433801 (+722470us) raft_consensus.cc:1685] finished
0430 07:59:12.433825 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:12.433973 (+ 148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":83047,"prepare.queue_time_us":68259,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":17164,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":34102,"spinlock_wait_cycles":108800,"thread_start_us":508383,"threads_started":2,"wal-append.queue_time_us":402949}
I20260430 07:59:12.436771 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.392045 9938 raft_consensus.cc:1275] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260430 07:59:12.405431 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:12.458305 10003 raft_consensus.cc:2368] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:12.407063 11850 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:12.407325 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:12.383503 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.472220 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:12.407661 11501 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:12.408070 10010 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:12.409565 11626 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:11.958987 12076 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:11.990118 12152 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:12.419414 12150 log.cc:927] Time spent T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.384s user 0.000s sys 0.001s
I20260430 07:59:12.511621 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.417187 11807 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:11.958076 11724 raft_consensus.cc:2804] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:59:11.809657 11995 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:12.385562 12003 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260430 07:59:12.420893 12060 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:11.990842 11769 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.386072 12051 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:12.422878 12146 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:59:12.423818 12149 raft_consensus.cc:3060] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:12.424592 10559 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:12.426615 11722 consensus_queue.cc:1059] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:11.923834 12078 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:12.428898 12054 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:12.429234 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:11.996977 11802 consensus_queue.cc:1059] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.433s
I20260430 07:59:12.429688 12052 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:12.163338 11903 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:12.208428 10562 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54778 (request call id 114) took 2114 ms. Trace:
I20260430 07:59:12.544042 12060 raft_consensus.cc:697] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:11.958866 11735 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:12.430444 12065 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.214269 12050 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:12.431578 11952 consensus_queue.cc:1059] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:12.436929 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:12.626729 9855 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 7
I20260430 07:59:12.438324 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.629104 10007 raft_consensus.cc:2410] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260430 07:59:11.984179 12137 log.cc:927] Time spent T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20260430 07:59:12.461737 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 999) took 1309 ms. Trace:
I20260430 07:59:12.471308 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.473263 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.631229 9938 rpcz_store.cc:276] 0430 07:59:11.152807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:12.167954 (+1015147us) service_pool.cc:225] Handling call
0430 07:59:12.168841 (+ 887us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:12.460526 (+291685us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:12.461625 (+ 1099us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:12.632464 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:12.633906 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:12.444691 11841 consensus_queue.cc:1048] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260430 07:59:12.544181 12064 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.007s sys 0.017s
W20260430 07:59:12.494866 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:12.500352 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:12.414772 12141 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.199678 11857 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260430 07:59:12.512535 10011 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.012s sys 0.010s
I20260430 07:59:12.651338 10004 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:12.501973 9978 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:12.502728 9902 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.114s user 0.026s sys 0.055s
I20260430 07:59:12.517242 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:11.990319 12142 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:12.518203 9857 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.000s sys 0.035s
W20260430 07:59:12.655321 9857 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.869s user 0.001s sys 0.035s
I20260430 07:59:12.590279 12146 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:12.530252 9903 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.007s sys 0.018s
I20260430 07:59:12.532400 9862 raft_consensus.cc:1275] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260430 07:59:12.670290 9903 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.753s user 0.007s sys 0.019s
I20260430 07:59:12.534296 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:12.670761 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 754375us
W20260430 07:59:12.534729 9932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 960) took 3679 ms (client timeout 1887 ms). Trace:
W20260430 07:59:12.671494 9932 rpcz_store.cc:269] 0430 07:59:08.855836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.700516 (+844680us) service_pool.cc:225] Handling call
0430 07:59:12.534406 (+2833890us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540027e7d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:12.534613 (+ 207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337571712}
I20260430 07:59:12.672031 9903 leader_election.cc:400] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:12.673072 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:12.591922 11807 raft_consensus.cc:3055] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
W20260430 07:59:12.537307 9937 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 983) took 2626 ms (client timeout 2424 ms). Trace:
W20260430 07:59:12.695082 9937 rpcz_store.cc:269] 0430 07:59:09.911188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.518017 (+606829us) service_pool.cc:225] Handling call
0430 07:59:12.537053 (+2019036us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b54002f1054. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:12.537207 (+ 154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87222144}
W20260430 07:59:12.540268 9933 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 978) took 2640 ms (client timeout 2736 ms). Trace:
W20260430 07:59:12.537905 10531 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.007s sys 0.016s
W20260430 07:59:12.697686 10531 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s user 0.008s sys 0.016s
W20260430 07:59:12.697481 9933 rpcz_store.cc:269] 0430 07:59:09.900064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.523022 (+622958us) service_pool.cc:225] Handling call
0430 07:59:12.540006 (+2016984us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540003c354. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:12.540192 (+ 186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337117568}
I20260430 07:59:11.969880 12144 consensus_queue.cc:1059] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.612s
W20260430 07:59:12.537959 10534 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.708s user 0.019s sys 0.004s
W20260430 07:59:12.707069 10534 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.878s user 0.019s sys 0.005s
W20260430 07:59:12.707510 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 894764us
I20260430 07:59:12.708429 10534 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:12.537767 11655 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.000s sys 0.023s
I20260430 07:59:12.518477 12076 consensus_queue.cc:1048] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.032s
W20260430 07:59:12.560369 12133 consensus_meta.cc:287] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 1.402s user 0.014s sys 0.000s
I20260430 07:59:12.559968 12051 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:12.717389 12144 consensus_queue.cc:1243] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer d66db415f0f0402fa431349afbf59888 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260430 07:59:12.717603 9863 raft_consensus.cc:2410] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260430 07:59:12.718761 12051 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:12.493387 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 123891us
I20260430 07:59:12.559639 11769 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:12.719717 11769 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:12.560005 12013 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:12.637558 10010 raft_consensus.cc:2410] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:12.725663 12013 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:12.727036 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.727742 10010 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:12.728785 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.729465 12054 consensus_queue.cc:1048] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:12.530547 12152 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:12.723794 9938 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:12.731982 12152 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.639070 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.734052 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:12.722594 11626 consensus_queue.cc:1048] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260430 07:59:12.644404 9929 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:12.736613 9855 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:12.636478 11735 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:12.646366 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:12.742624 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:12.646574 12060 consensus_queue.cc:237] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:12.647874 11789 raft_consensus.cc:3055] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:59:12.648666 11952 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:12.649250 11841 raft_consensus.cc:3055] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:12.637159 12050 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:12.651311 10011 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.840s user 0.013s sys 0.010s
W20260430 07:59:12.652849 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 151274us
W20260430 07:59:12.653010 9902 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.266s user 0.026s sys 0.055s
I20260430 07:59:12.653378 9936 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:12.674145 9932 raft_consensus.cc:2393] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:12.535218 9930 raft_consensus.cc:1275] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:12.681814 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.697093 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:12.664999 11722 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:12.715663 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:12.715703 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:12.712056 11655 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.875s user 0.000s sys 0.024s
I20260430 07:59:12.722784 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:12.737313 10534 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:12.589951 12078 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:12.739145 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:12.721083 9934 consensus_queue.cc:260] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:12.665665 11802 raft_consensus.cc:3055] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:12.656381 11857 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:12.528168 10005 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:12.591650 12153 consensus_queue.cc:1059] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.148s
I20260430 07:59:12.689808 12142 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:12.748950 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:13.050232 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:13.050366 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1670) took 1054 ms. Trace:
I20260430 07:59:13.050776 10005 rpcz_store.cc:276] 0430 07:59:11.996405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:12.199208 (+202803us) service_pool.cc:225] Handling call
0430 07:59:13.050305 (+851097us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.051196 9937 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
W20260430 07:59:12.784530 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:13.051755 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 328433us
I20260430 07:59:12.721778 10561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54770 (request call id 352) took 2044 ms. Trace:
I20260430 07:59:13.052510 9862 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:13.052898 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:13.053655 9859 raft_consensus.cc:2393] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
W20260430 07:59:12.645503 12064 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.647s user 0.009s sys 0.020s
I20260430 07:59:13.054010 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.054780 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.052692 9977 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:13.055797 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:12.734043 12157 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:13.057091 11952 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.058213 9934 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:13.049914 10006 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
W20260430 07:59:13.060638 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:12.869236 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:12.749845 9933 raft_consensus.cc:1275] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260430 07:59:13.061887 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:13.053001 10561 rpcz_store.cc:276] 0430 07:59:10.677707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.679192 (+ 1485us) service_pool.cc:225] Handling call
0430 07:59:10.753533 (+ 74341us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:10.804703 (+ 51170us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:12.705697 (+1900994us) spinlock_profiling.cc:243] Waited 49.3 ms on lock 0x7b5400430154. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:12.721572 (+ 15875us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":113388800}
W20260430 07:59:12.927673 12000 consensus_meta.cc:287] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 1.328s user 0.008s sys 0.005s
I20260430 07:59:13.063611 12000 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.066562 12000 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.943460 12156 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:12.721839 12065 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:13.134999 12051 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:13.069087 12078 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.008127 11724 raft_consensus.cc:697] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:12.844380 9929 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:12.902172 12003 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:13.071473 11802 raft_consensus.cc:740] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:12.591506 10562 rpcz_store.cc:276] 0430 07:59:10.048411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.055311 (+ 6900us) service_pool.cc:225] Handling call
0430 07:59:12.033183 (+1977872us) spinlock_profiling.cc:243] Waited 587 ms on lock 0x7b540047e0d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:12.162673 (+129490us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1350796672}
W20260430 07:59:13.072930 9854 consensus_meta.cc:287] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.962s user 0.011s sys 0.000s
W20260430 07:59:13.073994 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 336743us
I20260430 07:59:13.074100 9859 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: [4.3-5.4]
I20260430 07:59:13.243000 9854 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:13.243597 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1509) took 1341 ms. Trace:
I20260430 07:59:13.009171 11601 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:13.059352 11952 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.831104 12133 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:12.694762 11807 raft_consensus.cc:740] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:13.080533 9977 leader_election.cc:400] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:13.049882 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.057968 12157 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:12.722792 12141 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:12.741600 12015 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:59:13.164367 12156 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.255123 12159 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260430 07:59:13.283051 11859 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:12.722450 11850 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.361s
W20260430 07:59:13.156908 11826 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.003s sys 0.017s
I20260430 07:59:13.084995 12052 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:59:13.242998 12065 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.092729 12050 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.092574 11626 consensus_queue.cc:1059] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260430 07:59:13.311918 12157 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.707610 12146 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.314714 12146 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:12.887725 12154 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:13.315568 12154 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.318341 12154 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.050796 11789 raft_consensus.cc:740] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:13.320214 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 988) took 3219 ms. Trace:
I20260430 07:59:13.319661 11789 consensus_queue.cc:260] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:12.927213 12152 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:13.320783 9931 rpcz_store.cc:276] 0430 07:59:10.100430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.528438 (+1428008us) service_pool.cc:225] Handling call
0430 07:59:11.528791 (+ 353us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:11.557514 (+ 28723us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.319782 (+1762268us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b5400348354. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:13.320110 (+ 328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37577088}
I20260430 07:59:13.111224 12013 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.047297 11769 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.084547 11995 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:13.115660 9936 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:13.084211 12054 consensus_queue.cc:1059] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
W20260430 07:59:13.329808 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 251425us
I20260430 07:59:13.125140 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.049373 11722 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:13.044714 11748 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.048s user 0.048s sys 0.052s
I20260430 07:59:13.072278 11841 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:13.078967 12070 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:13.339967 9931 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260430 07:59:13.137398 10533 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.009s sys 0.010s
I20260430 07:59:13.159653 12166 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:13.243916 9854 rpcz_store.cc:276] 0430 07:59:11.902167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.903278 (+ 1111us) service_pool.cc:225] Handling call
0430 07:59:13.243532 (+1340254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.290943 12156 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.246709 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.244974 12078 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:13.249303 12047 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:13.252580 11903 consensus_meta.cc:287] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.615s user 0.005s sys 0.001s
I20260430 07:59:13.255299 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.247752 12051 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.348901 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:13.349772 10003 raft_consensus.cc:2368] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:13.264318 11724 consensus_queue.cc:237] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:13.272104 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 191669us
I20260430 07:59:13.275455 12141 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.276332 12033 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:13.278586 12142 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.287160 9934 raft_consensus.cc:1275] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260430 07:59:13.091475 11735 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:13.393054 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:13.399712 9861 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.011s sys 0.035s
W20260430 07:59:13.400367 9861 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s user 0.011s sys 0.035s
I20260430 07:59:13.401692 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1522) took 1031 ms. Trace:
W20260430 07:59:13.288599 9860 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.003s sys 0.010s
I20260430 07:59:13.308528 11807 consensus_queue.cc:260] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.274551 12133 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:13.309959 12149 consensus_meta.cc:287] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.723s user 0.007s sys 0.004s
I20260430 07:59:13.283620 11859 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:13.326812 9904 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.817s user 0.010s sys 0.011s
W20260430 07:59:13.467226 9904 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.957s user 0.010s sys 0.011s
W20260430 07:59:13.467636 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 957379us
I20260430 07:59:13.318826 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:13.354947 9934 pending_rounds.cc:85] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 5
I20260430 07:59:13.332137 9862 raft_consensus.cc:2393] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:13.335424 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.469090 10007 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:13.469072 9934 pending_rounds.cc:107] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260430 07:59:13.470150 9934 raft_consensus.cc:2889] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:13.470923 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 247) took 2334 ms. Trace:
W20260430 07:59:13.472087 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 121387us
I20260430 07:59:13.473486 9904 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:13.474226 9828 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.043s user 0.032s sys 0.092s
W20260430 07:59:13.474550 9828 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.043s user 0.032s sys 0.092s
W20260430 07:59:13.474872 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 1.04323e+06us
I20260430 07:59:13.475378 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.323796 11802 consensus_queue.cc:260] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.478822 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.480265 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.344967 12166 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.080312 11857 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:13.351276 12078 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.257491 12003 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.321516 12152 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.328059 12146 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:13.338516 11748 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.342s user 0.048s sys 0.052s
W20260430 07:59:13.289551 11826 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s user 0.003s sys 0.017s
I20260430 07:59:13.346342 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1645) took 3745 ms. Trace:
I20260430 07:59:13.346457 9856 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:13.500524 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1521) took 1130 ms. Trace:
I20260430 07:59:13.500859 9856 rpcz_store.cc:276] 0430 07:59:12.369755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.162310 (+792555us) service_pool.cc:225] Handling call
0430 07:59:13.500452 (+338142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.329932 11769 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:13.263016 11601 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.351660 9977 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:13.507347 11769 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.338073 11722 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:13.328498 12013 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:13.340368 11995 raft_consensus.cc:3055] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
W20260430 07:59:13.440501 9860 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.852s user 0.003s sys 0.010s
I20260430 07:59:13.481009 10004 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:13.471704 9934 rpcz_store.cc:276] 0430 07:59:11.136576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.903461 (+766885us) service_pool.cc:225] Handling call
0430 07:59:11.903731 (+ 270us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:12.720712 (+816981us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b5400185220. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb56920144f 00007fb56927a340 00007fb56927bbbe 00007fb569285f29 00007fb5692867f6 00007fb56928316f 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2
0430 07:59:13.470692 (+749980us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.470872 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1164278144}
I20260430 07:59:13.533561 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.416369 11841 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.353562 12047 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:13.341141 10533 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.921s user 0.010s sys 0.011s
I20260430 07:59:13.482072 12004 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260430 07:59:13.482657 12144 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260430 07:59:13.473838 12149 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:13.549386 11841 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:13.554740 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:13.557744 12141 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:13.558730 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 207087us
I20260430 07:59:13.494673 10007 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:13.356103 11903 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.543085 12004 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:13.561699 12004 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:13.560719 11826 consensus_queue.cc:1059] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:59:13.562150 11859 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:13.499970 10011 rpcz_store.cc:276] 0430 07:59:09.600477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:09.601382 (+ 905us) service_pool.cc:225] Handling call
0430 07:59:09.601831 (+ 449us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:11.411826 (+1809995us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:11.411859 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:11.446259 (+ 34400us) log.cc:844] Serialized 26 byte log entry
0430 07:59:11.787940 (+341681us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:13.048523 (+1260583us) log.cc:844] Serialized 14 byte log entry
0430 07:59:13.345889 (+297366us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.346011 (+ 122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:13.346077 (+ 66us) raft_consensus.cc:1685] finished
0430 07:59:13.346097 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:13.346270 (+ 173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1932263,"prepare.queue_time_us":852824,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":1820172,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":66936,"thread_start_us":1797454,"threads_started":3,"wal-append.queue_time_us":479913,"wal-append.run_cpu_time_us":2708,"wal-append.run_wall_time_us":1028030}
I20260430 07:59:13.487555 11807 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:13.512079 9978 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:13.345602 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:13.566221 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.567804 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:13.568233 12003 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:13.566226 11903 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.531734 11995 raft_consensus.cc:740] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:13.494755 11802 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:13.483965 9904 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:13.537395 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.570616 11995 consensus_queue.cc:260] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.571215 10004 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:13.571625 11995 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:13.572094 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.572742 10004 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:13.493474 12054 consensus_queue.cc:1059] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260430 07:59:13.573614 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 313) took 1819 ms. Trace:
I20260430 07:59:13.573802 12054 raft_consensus.cc:3055] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:13.574375 12054 raft_consensus.cc:740] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:13.575287 11769 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.574186 9857 rpcz_store.cc:276] 0430 07:59:11.754304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.754765 (+ 461us) service_pool.cc:225] Handling call
0430 07:59:11.784021 (+ 29256us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:12.655730 (+871709us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:12.655767 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:12.694249 (+ 38482us) log.cc:844] Serialized 26 byte log entry
0430 07:59:13.288339 (+594090us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:13.288388 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.288475 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:13.573357 (+284882us) raft_consensus.cc:1685] finished
0430 07:59:13.573379 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:13.573554 (+ 175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13618,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":5729,"thread_start_us":1319526,"threads_started":2,"wal-append.queue_time_us":829782}
I20260430 07:59:13.534924 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:13.539690 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 1.24964e+06us
I20260430 07:59:13.536545 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:13.564836 9977 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:13.566431 12133 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.535137 12013 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.542166 11655 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:13.580142 9977 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:13.582392 12013 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.582604 12152 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.582536 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:13.584225 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:13.584331 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.570153 12078 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:13.585798 11655 raft_consensus.cc:2764] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260430 07:59:13.589417 12181 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:13.590055 9977 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:13.590893 12047 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.575469 12054 consensus_queue.cc:260] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.500192 12146 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.592283 12051 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:13.521682 12156 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
W20260430 07:59:13.593775 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:13.594110 12065 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:13.596608 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.569056 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.606137 10011 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:13.529914 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.598253 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:13.598863 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 56071.2us
I20260430 07:59:13.596987 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:13.607874 10003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1672) took 1484 ms. Trace:
I20260430 07:59:13.607954 9856 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:13.608083 10004 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:59:13.601624 10533 connection.cc:254] RPC call timeout handler was delayed by -1.017276221999964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2073s for a response
I20260430 07:59:13.608295 10003 rpcz_store.cc:276] 0430 07:59:12.123932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.485584 (+1361652us) service_pool.cc:225] Handling call
0430 07:59:13.607815 (+122231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.610033 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:13.610706 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:13.612274 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 128363us
I20260430 07:59:13.610733 9934 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
W20260430 07:59:13.617668 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 105646us
I20260430 07:59:13.626353 9978 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:13.632560 9903 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:13.634371 10533 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:13.638257 10007 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:13.636760 10533 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:13.596655 11826 raft_consensus.cc:3055] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:13.643785 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.645352 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:13.646243 10003 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:13.648387 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.650367 9854 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:13.651590 9858 raft_consensus.cc:1240] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:13.643350 11769 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:13.625738 12004 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.654906 12004 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.657475 12054 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:13.657722 11857 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.660481 11857 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:13.662130 12049 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:13.662865 12146 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.663724 12146 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:13.664285 12146 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:13.665098 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:13.666371 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:13.666904 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:13.667266 11859 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:13.667721 11859 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:13.668107 11859 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:13.668926 11748 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:13.669097 12178 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:13.669803 12176 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:13.669986 11849 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:13.670365 12176 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.672617 12176 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.600984 11655 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:13.673409 11769 raft_consensus.cc:697] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:13.673761 11826 raft_consensus.cc:740] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:13.674378 11769 consensus_queue.cc:237] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.674625 11826 consensus_queue.cc:260] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.675962 11826 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:13.677407 12166 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.677525 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.678153 12049 raft_consensus.cc:697] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:13.679265 12049 consensus_queue.cc:237] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.679764 11601 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:13.623164 11626 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:13.668881 11625 raft_consensus.cc:2804] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:13.681602 11626 raft_consensus.cc:697] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:13.681939 11625 raft_consensus.cc:697] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [term 7 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:13.682605 11626 consensus_queue.cc:237] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.683164 11625 consensus_queue.cc:237] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.669800 12178 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.687669 12178 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:13.687889 11789 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:13.689997 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 219) took 1949 ms. Trace:
I20260430 07:59:13.691207 12070 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260430 07:59:13.692377 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.416450 9861 rpcz_store.cc:276] 0430 07:59:12.370043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:12.370195 (+ 152us) service_pool.cc:225] Handling call
0430 07:59:12.406105 (+ 35910us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:12.406335 (+ 230us) raft_consensus.cc:1534] Early marking committed up to index 3
0430 07:59:12.836619 (+430284us) log.cc:844] Serialized 14 byte log entry
0430 07:59:13.401130 (+564511us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:13.401218 (+ 88us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:13.401313 (+ 95us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.401416 (+ 103us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:13.401608 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17929,"prepare.queue_time_us":856088,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":137,"thread_start_us":975898,"threads_started":2}
I20260430 07:59:13.695160 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.696012 9861 raft_consensus.cc:2393] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:13.696861 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.669697 11748 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:13.714011 11748 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:13.715373 10004 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:13.677979 12157 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:59:13.699846 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.690510 10009 rpcz_store.cc:276] 0430 07:59:11.740064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.742329 (+ 2265us) service_pool.cc:225] Handling call
0430 07:59:11.742616 (+ 287us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:12.413547 (+670931us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:12.413575 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:12.414201 (+ 626us) log.cc:844] Serialized 26 byte log entry
0430 07:59:12.902499 (+488298us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:12.902541 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.689545 (+787004us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b540018ced4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:13.689600 (+ 55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:13.689653 (+ 53us) raft_consensus.cc:1685] finished
0430 07:59:13.689671 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:13.689928 (+ 257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":108,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"spinlock_wait_cycles":1541446528,"thread_start_us":487589,"threads_started":1,"wal-append.queue_time_us":744803}
I20260430 07:59:13.702418 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1384) took 5476 ms. Trace:
I20260430 07:59:13.717003 9935 rpcz_store.cc:276] 0430 07:59:08.226391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.130642 (+2904251us) service_pool.cc:225] Handling call
0430 07:59:11.414158 (+283516us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:11.556319 (+142161us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.074636 (+1518317us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b540036bfd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:13.702054 (+627418us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b540036bfd0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:13.702355 (+ 301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":953907328}
I20260430 07:59:13.717566 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.718470 10009 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:13.716624 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.719719 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:13.703819 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:13.705672 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:13.720651 10004 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:13.706921 9857 raft_consensus.cc:1240] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
W20260430 07:59:13.709018 10559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54778 (request call id 116) took 3389 ms (client timeout 1807 ms). Trace:
I20260430 07:59:13.709950 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 216) took 1970 ms. Trace:
I20260430 07:59:13.710080 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:13.719801 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:13.728519 10008 rpcz_store.cc:276] 0430 07:59:11.739210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.741916 (+ 2706us) service_pool.cc:225] Handling call
0430 07:59:11.742152 (+ 236us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:12.382082 (+639930us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:12.382110 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:12.382150 (+ 40us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:12.382180 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.709716 (+1327536us) spinlock_profiling.cc:243] Waited 639 ms on lock 0x7b54000cce54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:13.709759 (+ 43us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:13.709898 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1470731008}
I20260430 07:59:13.733083 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.734664 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.735388 10005 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:13.736366 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.736913 10005 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:59:13.740125 9856 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
W20260430 07:59:13.741732 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 113526us
I20260430 07:59:13.742775 10533 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:13.744760 10533 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:13.745828 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:13.747687 10004 raft_consensus.cc:2410] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:13.749639 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.750392 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.751461 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.752291 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.753950 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.755357 10004 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:13.756099 10004 raft_consensus.cc:606] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:13.756538 10004 raft_consensus.cc:614] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:13.742775 9977 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:13.757069 10004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:55676 (request call id 16) took 1344 ms. Trace:
I20260430 07:59:13.758693 10004 rpcz_store.cc:276] 0430 07:59:12.412928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.755204 (+1342276us) service_pool.cc:225] Handling call
0430 07:59:13.756992 (+ 1788us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.762053 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1564) took 1251 ms. Trace:
I20260430 07:59:13.762903 9860 rpcz_store.cc:276] 0430 07:59:12.510631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:12.510852 (+ 221us) service_pool.cc:225] Handling call
0430 07:59:12.515013 (+ 4161us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:12.515548 (+ 535us) raft_consensus.cc:1534] Early marking committed up to index 3
0430 07:59:13.534402 (+1018854us) log.cc:844] Serialized 14 byte log entry
0430 07:59:13.761541 (+227139us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:13.761624 (+ 83us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:13.761698 (+ 74us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.761796 (+ 98us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:13.761972 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":747064,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"thread_start_us":1138833,"threads_started":2}
I20260430 07:59:13.767463 10008 raft_consensus.cc:1240] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:13.769464 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:13.771929 10003 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
W20260430 07:59:13.772626 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 140619us
W20260430 07:59:13.728232 10559 rpcz_store.cc:269] 0430 07:59:10.320448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:10.340979 (+ 20531us) service_pool.cc:225] Handling call
0430 07:59:13.708820 (+3367841us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b540063a954. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:13.708946 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4689230592}
I20260430 07:59:13.775902 11827 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:13.762313 12013 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:13.802831 12013 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:13.803308 12013 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:13.803752 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:13.783695 10004 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:13.773358 12047 consensus_queue.cc:1048] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260430 07:59:13.818746 12146 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:13.795315 11857 consensus_queue.cc:1059] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260430 07:59:13.798081 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 998) took 1982 ms. Trace:
I20260430 07:59:13.762607 12159 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:13.765733 12181 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:59:13.783778 11655 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:13.789775 12015 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:13.806965 11804 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:13.764545 11722 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.808182 10007 raft_consensus.cc:1240] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:13.854687 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:13.856374 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:13.809782 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.811348 11850 consensus_queue.cc:1059] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:13.857996 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1674) took 1449 ms. Trace:
I20260430 07:59:13.857893 9931 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:13.858667 10007 rpcz_store.cc:276] 0430 07:59:12.408526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.807693 (+1399167us) service_pool.cc:225] Handling call
0430 07:59:13.808005 (+ 312us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:13.855215 (+ 47210us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.857913 (+ 2698us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.859700 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:13.860311 10007 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:13.860519 9931 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:13.830288 10008 rpcz_store.cc:276] 0430 07:59:11.785700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.766920 (+1981220us) service_pool.cc:225] Handling call
0430 07:59:13.767297 (+ 377us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:13.768001 (+ 704us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.768197 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:13.831148 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:13.861781 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:13.862321 9934 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:13.865959 10008 raft_consensus.cc:1240] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:13.867126 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1011) took 1374 ms. Trace:
I20260430 07:59:13.867594 10008 rpcz_store.cc:276] 0430 07:59:12.492500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.865121 (+1372621us) service_pool.cc:225] Handling call
0430 07:59:13.865524 (+ 403us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:13.866770 (+ 1246us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.867046 (+ 276us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:13.832949 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 63549.7us
I20260430 07:59:13.834941 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:13.838119 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 314479us
I20260430 07:59:13.840580 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:13.870913 9936 raft_consensus.cc:2393] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 7: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:13.870831 9937 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:13.843536 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:13.846567 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:13.872133 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.849798 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:13.872587 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.873435 9854 raft_consensus.cc:2393] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:13.872651 9858 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:13.875044 9938 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260430 07:59:13.874931 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.876425 10009 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:13.877532 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:13.878652 9828 connection.cc:254] RPC call timeout handler was delayed by -1.009275840999976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
I20260430 07:59:13.879956 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:13.880564 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:13.881242 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:13.882120 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:13.882345 9932 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:13.882810 9936 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:13.836795 11857 consensus_queue.cc:1243] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer 77e19ab653f540339c1c613fad1451b5 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260430 07:59:13.888069 9758 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:53852 (request call id 91) took 18640 ms (client timeout 15000 ms). Trace:
I20260430 07:59:13.892158 10005 raft_consensus.cc:2393] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:13.854733 12015 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:13.909900 10004 raft_consensus.cc:606] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:13.885032 11850 consensus_queue.cc:1243] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer 27f97a330f374849be58a168318c1ec6 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260430 07:59:13.888556 9758 rpcz_store.cc:269] 0430 07:58:55.242085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:55.242366 (+ 281us) service_pool.cc:225] Handling call
0430 07:59:06.920116 (+11677750us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b4000008278. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56f065170 00007fb56f1a7ba2 00007fb56f024537 00007fb56f129cae 00007fb56d60f4a4 00007fb56d60f432 00007fb56d60f361
0430 07:59:13.882111 (+6961995us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:59:09.612375 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:59:09.613114 (+ 739us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:59:09.613168 (+ 54us) write_op.cc:435] Acquired schema lock
0430 07:59:09.613190 (+ 22us) tablet.cc:662] Decoding operations
0430 07:59:09.622070 (+ 8880us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:59:09.622187 (+ 117us) write_op.cc:641] Partition lock acquired for write op
0430 07:59:09.622230 (+ 43us) tablet.cc:685] Acquiring locks for 5 operations
0430 07:59:09.623143 (+ 913us) tablet.cc:701] Row locks acquired
0430 07:59:09.623178 (+ 35us) write_op.cc:260] PREPARE: finished
0430 07:59:09.623426 (+ 248us) write_op.cc:270] Start()
0430 07:59:09.623617 (+ 191us) write_op.cc:276] Timestamp: P: 1777535949623325 usec, L: 0
0430 07:59:09.623666 (+ 49us) op_driver.cc:348] REPLICATION: starting
0430 07:59:09.748801 (+125135us) log.cc:844] Serialized 2064 byte log entry
0430 07:59:10.393503 (+644702us) op_driver.cc:464] REPLICATION: finished
0430 07:59:10.415607 (+ 22104us) write_op.cc:301] APPLY: starting
0430 07:59:10.415729 (+ 122us) tablet.cc:1367] starting BulkCheckPresence
0430 07:59:10.415968 (+ 239us) tablet.cc:1370] finished BulkCheckPresence
0430 07:59:10.416158 (+ 190us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:59:10.518982 (+102824us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:59:10.519376 (+ 394us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0430 07:59:10.519420 (+ 44us) write_op.cc:312] APPLY: finished
0430 07:59:10.519735 (+ 315us) log.cc:844] Serialized 46 byte log entry
0430 07:59:10.519956 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:59:10.520071 (+ 115us) write_op.cc:454] Released schema lock
0430 07:59:10.520449 (+ 378us) write_op.cc:341] FINISH: Updating metrics
0430 07:59:10.525923 (+ 5474us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:59:10.525957 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1089878,"reported_tablets":16,"spinlock_wait_cycles":4026624,"child_traces":[["op",{"apply.queue_time_us":21777,"apply.run_cpu_time_us":4656,"apply.run_wall_time_us":110291,"num_ops":5,"prepare.queue_time_us":574305,"prepare.run_cpu_time_us":35687,"prepare.run_wall_time_us":364840,"raft.queue_time_us":309277,"raft.run_cpu_time_us":5573,"raft.run_wall_time_us":61766,"replication_time_us":749974,"thread_start_us":741728,"threads_started":4,"wal-append.queue_time_us":269786,"wal-append.run_cpu_time_us":76560,"wal-append.run_wall_time_us":1525450}]]}
I20260430 07:59:13.885735 12051 raft_consensus.cc:476] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Not starting pre-election -- already a leader
I20260430 07:59:13.907328 12015 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:13.882040 9937 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:13.886521 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1010) took 1411 ms. Trace:
W20260430 07:59:13.924856 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 92901us
I20260430 07:59:13.925346 10007 rpcz_store.cc:276] 0430 07:59:12.449920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.859691 (+1409771us) service_pool.cc:225] Handling call
0430 07:59:13.860056 (+ 365us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:13.860869 (+ 813us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.861236 (+ 367us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.926805 10007 raft_consensus.cc:1275] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:13.927438 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1013) took 1388 ms. Trace:
I20260430 07:59:13.927750 10007 rpcz_store.cc:276] 0430 07:59:12.538964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.926168 (+1387204us) service_pool.cc:225] Handling call
0430 07:59:13.926478 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:13.927194 (+ 716us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.927386 (+ 192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.834815 12159 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:13.919514 10009 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:13.848780 12012 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:13.879747 11804 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:13.924582 9858 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:13.929082 10007 raft_consensus.cc:1275] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:13.929205 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:13.955418 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 146591us
I20260430 07:59:13.886080 10008 raft_consensus.cc:1275] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:13.937870 9931 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:13.938077 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 305468us
I20260430 07:59:13.949523 9828 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:13.958292 9904 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:13.924324 12051 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:13.963685 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1014) took 1421 ms. Trace:
I20260430 07:59:13.969882 10007 rpcz_store.cc:276] 0430 07:59:12.542268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.928440 (+1386172us) service_pool.cc:225] Handling call
0430 07:59:13.928712 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:13.954985 (+ 26273us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.963567 (+ 8582us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.966715 9856 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:13.971866 10007 raft_consensus.cc:1275] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260430 07:59:13.967824 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:13.961774 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1012) took 1447 ms. Trace:
I20260430 07:59:13.945747 12159 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:14.020429 10005 raft_consensus.cc:1275] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:14.026001 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1019) took 1278 ms. Trace:
I20260430 07:59:14.025851 12051 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.026680 10005 rpcz_store.cc:276] 0430 07:59:12.748401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:14.019443 (+1271042us) service_pool.cc:225] Handling call
0430 07:59:14.019830 (+ 387us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:14.021136 (+ 1306us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:14.025917 (+ 4781us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.959812 12000 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:14.032521 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 981) took 4122 ms. Trace:
W20260430 07:59:13.976698 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:13.977229 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:14.032929 9930 rpcz_store.cc:276] 0430 07:59:09.910022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.502123 (+1592101us) service_pool.cc:225] Handling call
0430 07:59:12.534886 (+1032763us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:13.048830 (+513944us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:14.032324 (+983494us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540027e7d0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:14.032465 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2343007104}
I20260430 07:59:13.977309 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:13.969112 11841 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:13.978850 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:13.981860 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:13.956324 12012 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:14.036134 12181 raft_consensus.cc:3055] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:14.036440 12012 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:14.036638 12181 raft_consensus.cc:740] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:14.037596 12181 consensus_queue.cc:260] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.038806 12181 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:14.039445 11827 raft_consensus.cc:3055] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:14.039894 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:44006 (request call id 543) took 1581 ms. Trace:
I20260430 07:59:13.965780 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:13.985617 12188 log.cc:927] Time spent T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20260430 07:59:13.968199 9903 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:13.985842 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:13.987033 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:13.986346 12185 log.cc:927] Time spent T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260430 07:59:13.987783 11722 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:13.982527 11859 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.040366 10009 rpcz_store.cc:276] 0430 07:59:12.458441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:14.038773 (+1580332us) service_pool.cc:225] Handling call
0430 07:59:14.039826 (+ 1053us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:14.045306 10011 raft_consensus.cc:2393] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:14.045643 11859 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:14.045926 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:13.990541 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:14.047663 10006 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:14.048409 10006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1678) took 1321 ms. Trace:
I20260430 07:59:13.992671 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:14.050017 9856 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:14.051991 11804 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.054680 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:13.993446 11986 raft_consensus.cc:3055] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:14.055328 10010 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:14.055481 11986 raft_consensus.cc:740] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:14.055321 9861 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:14.056478 11986 consensus_queue.cc:260] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.057391 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1017) took 1466 ms. Trace:
I20260430 07:59:14.057919 11986 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:14.057802 10007 rpcz_store.cc:276] 0430 07:59:12.621135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.971016 (+1349881us) service_pool.cc:225] Handling call
0430 07:59:13.971419 (+ 403us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:13.972396 (+ 977us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:14.057327 (+ 84931us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:13.976769 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:13.991366 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:14.002452 12189 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260430 07:59:14.004243 9863 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41094 (request call id 309) took 2456 ms (client timeout 2073 ms). Trace:
I20260430 07:59:14.062846 12189 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:14.016534 12076 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:14.063339 12189 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:14.063076 9863 rpcz_store.cc:269] 0430 07:59:11.547929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:11.548104 (+ 175us) service_pool.cc:225] Handling call
0430 07:59:14.003919 (+2455815us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54001bab54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:14.004138 (+ 219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2618959616}
I20260430 07:59:14.065760 9858 raft_consensus.cc:1240] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:14.016861 10004 raft_consensus.cc:1275] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260430 07:59:14.066650 12051 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.066715 9861 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:14.067318 10004 pending_rounds.cc:85] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 3
I20260430 07:59:14.067698 10004 pending_rounds.cc:107] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260430 07:59:14.067932 10004 raft_consensus.cc:2889] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:14.032172 12000 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.039853 11827 raft_consensus.cc:740] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:14.070428 11827 consensus_queue.cc:260] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.071580 12000 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.071811 11827 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:13.988710 12180 consensus_queue.cc:1059] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:59:14.047487 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 1020) took 1153 ms. Trace:
I20260430 07:59:13.991340 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:14.073751 12180 raft_consensus.cc:3055] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:14.074252 12180 raft_consensus.cc:740] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:14.075070 12180 consensus_queue.cc:260] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.076123 12180 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:14.048753 10006 rpcz_store.cc:276] 0430 07:59:12.726772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.061341 (+334569us) service_pool.cc:225] Handling call
0430 07:59:14.045731 (+984390us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b58000a2098. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:14.048156 (+ 2425us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28495232}
W20260430 07:59:14.059638 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 82951.9us
I20260430 07:59:14.056608 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 1021) took 1124 ms. Trace:
I20260430 07:59:14.079986 10010 rpcz_store.cc:276] 0430 07:59:12.932452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.054332 (+121880us) service_pool.cc:225] Handling call
0430 07:59:13.809617 (+755285us) spinlock_profiling.cc:243] Waited 26.4 ms on lock 0x7b58000a2098. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:14.056438 (+246821us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b5400147154. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:14.056550 (+ 112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":514927872}
W20260430 07:59:14.078603 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 109958us
I20260430 07:59:14.062217 11841 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:14.073108 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.073583 10012 rpcz_store.cc:276] 0430 07:59:12.893890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.049361 (+155471us) service_pool.cc:225] Handling call
0430 07:59:13.990345 (+940984us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b58000a2098. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:14.047405 (+ 57060us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":76056832}
I20260430 07:59:14.078418 9828 leader_election.cc:400] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:14.087515 9828 leader_election.cc:403] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260430 07:59:14.090451 9828 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:59:14.092834 9828 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:13.974004 10008 rpcz_store.cc:276] 0430 07:59:12.521591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.883997 (+1362406us) service_pool.cc:225] Handling call
0430 07:59:13.884485 (+ 488us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:13.961366 (+ 76881us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:13.961697 (+ 331us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:14.068585 10004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1676) took 1343 ms. Trace:
I20260430 07:59:14.073593 12076 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:14.099836 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 229867us
I20260430 07:59:14.094167 10004 rpcz_store.cc:276] 0430 07:59:12.725087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:14.016123 (+1291036us) service_pool.cc:225] Handling call
0430 07:59:14.016480 (+ 357us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:14.068379 (+ 51899us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:14.068535 (+ 156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:14.078580 9978 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:14.125106 10533 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:14.127242 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 164451us
I20260430 07:59:14.127745 10533 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:14.130685 9903 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:14.141484 11804 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:14.117432 12000 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:14.145712 12000 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:14.146173 12000 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:14.147425 12189 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.144430 12013 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:14.149148 12149 raft_consensus.cc:2749] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:59:14.149428 12049 raft_consensus.cc:2804] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:14.150012 12049 raft_consensus.cc:697] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:14.150053 12070 raft_consensus.cc:2804] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:14.150427 12013 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.150741 12070 raft_consensus.cc:697] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:14.151401 10565 raft_consensus.cc:2393] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 7: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:59:14.150985 12049 consensus_queue.cc:237] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:14.151660 12070 consensus_queue.cc:237] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:14.120375 11995 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:14.153717 11995 raft_consensus.cc:697] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:14.154599 11995 consensus_queue.cc:237] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.101061 12051 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:14.157547 12051 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:14.157980 12051 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:14.159651 9904 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:14.095021 12182 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:14.148433 9903 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:14.168617 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.177773 9903 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:14.190338 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:14.148440 12076 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:14.191897 9977 connection.cc:254] RPC call timeout handler was delayed by -1.0345365549999883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15100000000000002s for a response
I20260430 07:59:14.173334 11769 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:14.188961 12189 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.192776 12196 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:14.199308 12196 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.205780 12196 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:14.206409 12159 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:14.195636 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:14.180454 11849 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:14.218636 10568 raft_consensus.cc:2368] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:14.157582 12015 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:14.231191 10563 raft_consensus.cc:2368] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:14.208981 12159 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.219518 11769 raft_consensus.cc:697] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:14.261919 11769 consensus_queue.cc:237] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.223059 12000 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:14.242355 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.266372 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:14.267784 12000 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.210430 12012 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:14.284328 12012 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.290812 12015 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.318056 12051 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:14.320281 12051 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:14.095834 9828 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:14.098040 9856 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
W20260430 07:59:14.381916 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 285900us
I20260430 07:59:14.383648 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:14.386284 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.391009 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:14.392383 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:14.392520 9860 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:14.396678 9863 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260430 07:59:14.392383 9861 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:14.401111 9858 raft_consensus.cc:1275] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:14.401988 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:14.409734 9860 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:14.402495 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:14.403079 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:14.405030 9828 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:14.417498 9863 pending_rounds.cc:85] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 5
I20260430 07:59:14.418889 9863 pending_rounds.cc:107] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260430 07:59:14.427030 9863 raft_consensus.cc:2889] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:14.417543 9828 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:14.440419 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:14.443312 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 265433us
W20260430 07:59:14.444199 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.444809 9903 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:14.447757 9903 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:14.456123 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.426440 9860 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:14.463052 9904 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:14.492923 10003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1662) took 2681 ms. Trace:
I20260430 07:59:14.493773 10003 rpcz_store.cc:276] 0430 07:59:11.811120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.800249 (+1989129us) service_pool.cc:225] Handling call
0430 07:59:13.800560 (+ 311us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:13.800758 (+ 198us) raft_consensus.cc:1534] Early marking committed up to index 7
0430 07:59:13.976904 (+176146us) log.cc:844] Serialized 14 byte log entry
0430 07:59:14.295324 (+318420us) log.cc:844] Serialized 14 byte log entry
0430 07:59:14.491919 (+196595us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:14.492049 (+ 130us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:14.492183 (+ 134us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:14.492354 (+ 171us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:14.492622 (+ 268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":198771,"prepare.queue_time_us":414029,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":110,"raft.queue_time_us":19536,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":5277,"thread_start_us":432072,"threads_started":2,"wal-append.queue_time_us":353269}
I20260430 07:59:14.466760 9860 raft_consensus.cc:1275] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
W20260430 07:59:14.467532 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:14.489741 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.501050 10012 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:14.501641 9978 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:14.502079 10012 raft_consensus.cc:606] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
W20260430 07:59:14.512363 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 196261us
I20260430 07:59:14.519367 10012 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:14.520538 10012 raft_consensus.cc:606] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:14.525871 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:14.527391 10005 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260430 07:59:14.530184 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:14.531044 10003 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:14.532096 10012 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260430 07:59:14.533557 10005 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:14.534336 10005 raft_consensus.cc:606] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:14.534811 10005 raft_consensus.cc:614] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:14.534874 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:14.535914 9933 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:14.537804 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260430 07:59:14.544268 10534 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:14.571949 12157 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:14.574494 12157 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:14.578409 9760 catalog_manager.cc:5671] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 3 to 4, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 4 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:59:14.539479 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:14.596097 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.599159 9863 raft_consensus.cc:1240] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:14.586035 10012 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 6
I20260430 07:59:14.549813 11804 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:14.556190 11748 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:14.607172 11748 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:14.607615 11748 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:59:14.610599 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 158539us
I20260430 07:59:14.547436 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:59:14.538764 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:14.663877 10003 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:14.564082 12002 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:14.566645 12051 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader election won for term 3
I20260430 07:59:14.674960 9933 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:14.568125 12076 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:14.558166 12000 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:14.559420 11995 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:14.563016 11841 raft_consensus.cc:2764] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260430 07:59:14.559773 11655 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:14.556607 11789 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:14.715504 12076 consensus_queue.cc:1048] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:14.717725 11995 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:14.716122 11655 raft_consensus.cc:697] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:14.567601 12065 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:14.678634 12002 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:14.750852 11655 consensus_queue.cc:237] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.751791 12002 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:14.752324 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260430 07:59:14.731719 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.738497 9760 catalog_manager.cc:5671] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 1 to 2, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:14.720203 12051 raft_consensus.cc:697] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 3 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:14.760726 12051 consensus_queue.cc:237] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.762698 11995 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:14.764573 9903 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:14.746418 10012 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:14.747481 12182 consensus_queue.cc:1059] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260430 07:59:14.681565 12000 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:14.772320 12000 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:14.561965 11826 consensus_queue.cc:1059] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.192s
W20260430 07:59:14.775570 9930 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.039s sys 0.075s
I20260430 07:59:14.562641 11827 consensus_queue.cc:1048] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.204s
W20260430 07:59:14.787657 9930 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s user 0.039s sys 0.076s
I20260430 07:59:14.795104 12181 raft_consensus.cc:993] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: : Instructing follower 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:59:14.795838 12181 raft_consensus.cc:1081] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Signalling peer 27f97a330f374849be58a168318c1ec6 to start an election
W20260430 07:59:14.808723 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 324259us
W20260430 07:59:14.811403 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 348588us
W20260430 07:59:14.821508 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 282818us
I20260430 07:59:14.747509 12159 consensus_queue.cc:1059] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260430 07:59:14.809717 11789 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:14.855387 11789 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.859257 11789 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.861404 11769 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:14.861883 9855 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260430 07:59:14.862907 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:14.864732 12190 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:14.868500 9862 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07"
dest_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:37838
I20260430 07:59:14.863571 9860 raft_consensus.cc:1275] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:14.871449 9855 raft_consensus.cc:1240] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:14.869238 9862 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:59:14.882265 9862 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:14.890323 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 56068.9us
W20260430 07:59:14.891187 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.902673 11722 consensus_queue.cc:1048] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260430 07:59:14.869676 12197 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:14.886552 12159 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260430 07:59:14.911118 12189 consensus_queue.cc:1048] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260430 07:59:14.916823 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 247088us
I20260430 07:59:14.920511 11748 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.922889 11748 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.897334 12190 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:14.916611 9760 catalog_manager.cc:5671] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260430 07:59:14.926909 12002 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.927753 12190 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:14.927940 9760 catalog_manager.cc:5671] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:14.929091 12002 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.934877 12180 raft_consensus.cc:993] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: : Instructing follower 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:59:14.940346 9858 raft_consensus.cc:1240] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:14.938690 9903 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:14.945051 11769 consensus_queue.cc:1059] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260430 07:59:14.949607 11748 raft_consensus.cc:3055] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:14.939409 12065 raft_consensus.cc:3055] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:14.952143 12065 raft_consensus.cc:740] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:14.954231 12215 log.cc:927] Time spent T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260430 07:59:14.953478 12065 consensus_queue.cc:260] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.955252 12065 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:14.956429 12049 raft_consensus.cc:3055] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:14.956858 12049 raft_consensus.cc:740] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:14.908635 12182 consensus_queue.cc:1048] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260430 07:59:14.935544 11722 raft_consensus.cc:1081] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Signalling peer 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:59:14.957358 9760 catalog_manager.cc:5671] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:14.970233 11841 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:14.970388 11804 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:14.970989 11841 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:14.973646 11841 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:14.979326 9903 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:14.950055 11748 raft_consensus.cc:740] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:59:14.987808 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:14.964807 10533 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:59:15.032344 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 47304.1us
W20260430 07:59:15.040812 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 93620.1us
I20260430 07:59:15.036777 9862 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.046411 9855 raft_consensus.cc:1240] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
W20260430 07:59:15.047979 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 68879.4us
I20260430 07:59:15.049373 11804 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:15.049175 12223 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.099s
I20260430 07:59:15.053611 9903 leader_election.cc:400] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
W20260430 07:59:15.046348 9977 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.015s sys 0.024s
I20260430 07:59:15.062201 12202 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:15.063058 12202 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:15.070565 9977 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.854s user 0.016s sys 0.025s
I20260430 07:59:15.072525 9828 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:15.072755 9903 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:15.072299 12051 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:15.080778 9862 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:15.074594 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:15.048254 11748 consensus_queue.cc:260] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:15.066056 12159 raft_consensus.cc:2804] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:15.108050 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:14.970541 12049 consensus_queue.cc:260] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:15.111866 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:15.109853 9930 raft_consensus.cc:1275] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260430 07:59:15.101095 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 884205us
W20260430 07:59:15.119280 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.122478 9856 raft_consensus.cc:1275] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:15.123250 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:15.130174 9861 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:15.107241 12223 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:15.095722 11789 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:15.135147 12223 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:15.143728 12223 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:15.111321 12033 consensus_queue.cc:1059] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260430 07:59:15.146299 9854 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8"
dest_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:37838
I20260430 07:59:15.153421 9854 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:59:15.153803 9854 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:15.117311 12159 raft_consensus.cc:697] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:15.159240 9929 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260430 07:59:15.162246 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.118852 12202 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:15.132875 12219 consensus_queue.cc:1048] T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
W20260430 07:59:15.159253 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 86628.7us
I20260430 07:59:15.144212 12049 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:15.171916 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.146780 12033 raft_consensus.cc:3055] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:15.187480 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:15.184871 9760 catalog_manager.cc:5671] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 reported cstate change: term changed from 3 to 4, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 4 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } }
I20260430 07:59:15.213454 12196 raft_consensus.cc:3055] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:15.213979 12196 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:15.215076 12196 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.217262 12196 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:15.224471 12229 log.cc:927] Time spent T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.120s user 0.000s sys 0.001s
I20260430 07:59:15.185137 12033 raft_consensus.cc:740] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:15.229425 12033 consensus_queue.cc:260] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.230890 12033 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:15.181044 12000 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.235080 12000 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:15.235777 9860 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:15.240317 9828 leader_election.cc:400] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5.
W20260430 07:59:15.244854 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:15.246735 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.243258 9854 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.250365 9903 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:15.251490 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 49675.9us
W20260430 07:59:15.254284 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.260962 9854 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:15.190249 11789 consensus_queue.cc:1048] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.062s
I20260430 07:59:15.263051 9856 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:15.264573 12190 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:15.279090 10534 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:15.260591 12196 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:15.224547 12223 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:15.303766 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 125594us
W20260430 07:59:15.307195 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 45086.4us
W20260430 07:59:15.309479 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 107901us
I20260430 07:59:15.321331 12231 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260430 07:59:15.332993 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:15.314033 12012 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260430 07:59:15.347127 12070 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:15.348276 12223 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:15.359129 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 45934.2us
I20260430 07:59:15.322254 12196 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:15.344313 9977 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.842s (SENT) [suppressed 25 similar messages]
I20260430 07:59:15.360219 12070 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:15.364928 12126 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election won for term 4
W20260430 07:59:15.378583 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:15.383705 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 68802.4us
W20260430 07:59:15.390938 12225 log.cc:927] Time spent T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.095s user 0.000s sys 0.002s
W20260430 07:59:15.391549 9977 leader_election.cc:336] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.938s (SENT)
I20260430 07:59:15.319813 12159 consensus_queue.cc:237] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:15.396232 9859 raft_consensus.cc:1275] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:15.380107 12070 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:15.406395 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.389987 12126 raft_consensus.cc:697] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:15.404980 12203 consensus_queue.cc:1048] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260430 07:59:15.413610 9903 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:15.420233 9977 leader_election.cc:336] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.050s (SENT)
W20260430 07:59:15.428007 12213 log.cc:927] Time spent T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.667s user 0.007s sys 0.007s
W20260430 07:59:15.434463 12217 log.cc:927] Time spent T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.636s user 0.005s sys 0.006s
I20260430 07:59:15.434204 12126 consensus_queue.cc:237] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:15.438295 12203 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:15.439250 12203 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.442704 12203 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:15.454115 12196 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:15.454136 11769 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:15.472327 9977 leader_election.cc:336] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.229s (SENT)
W20260430 07:59:15.473682 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 67895.9us
W20260430 07:59:15.474215 9977 leader_election.cc:336] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.237s (SENT)
I20260430 07:59:15.474593 9903 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:15.475955 9903 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:15.477082 11769 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:15.478037 11769 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:15.478549 11769 raft_consensus.cc:697] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:15.479460 11769 consensus_queue.cc:237] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.483305 11995 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:15.484393 12189 consensus_queue.cc:1059] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:15.486490 12002 raft_consensus.cc:3055] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:15.452901 11748 raft_consensus.cc:3060] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:15.495281 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:15.495997 9856 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:15.500243 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:15.505941 9861 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
W20260430 07:59:15.511157 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.487007 12002 raft_consensus.cc:740] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:59:15.508359 12065 consensus_meta.cc:287] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.553s user 0.014s sys 0.002s
I20260430 07:59:15.542479 9860 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:15.546893 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:15.549147 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 48450.5us
I20260430 07:59:15.551404 9929 raft_consensus.cc:2393] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:59:15.547777 9904 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:15.502413 11995 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.507316 12196 consensus_queue.cc:1059] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
W20260430 07:59:15.564018 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 245789us
W20260430 07:59:15.567351 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:15.566001 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 82677.6us
I20260430 07:59:15.571871 9978 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:15.559051 11804 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:15.562008 11722 raft_consensus.cc:993] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: : Instructing follower 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:59:15.582273 9978 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:15.587445 12228 log.cc:927] Time spent T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.346s user 0.000s sys 0.007s
I20260430 07:59:15.591674 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:15.586472 11804 raft_consensus.cc:697] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:15.600982 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:15.609757 9929 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:15.610916 12239 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:15.611414 12239 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:15.611796 12239 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:15.611901 12070 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:15.613363 11789 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:15.614018 11789 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.614387 11804 consensus_queue.cc:237] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.615690 11655 raft_consensus.cc:3055] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:15.589143 12002 consensus_queue.cc:260] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.616461 12180 raft_consensus.cc:1081] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Signalling peer 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:59:15.617364 12070 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:15.610791 9903 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:15.618537 12180 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:15.619455 12190 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:15.616304 11655 raft_consensus.cc:740] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:15.602546 12196 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:15.622006 11655 consensus_queue.cc:260] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.623587 11655 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:15.620577 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:15.626863 10568 raft_consensus.cc:2393] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:59:15.629673 12180 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:15.630542 12180 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:15.619791 9933 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
W20260430 07:59:15.638283 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.656867 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:15.618638 12002 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:15.622319 12196 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.662909 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:15.670456 9859 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:15.670647 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 91096.3us
I20260430 07:59:15.670930 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:15.625110 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:15.666595 12197 consensus_queue.cc:1048] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260430 07:59:15.692138 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:15.693980 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.708606 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:15.718910 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 67427us
W20260430 07:59:15.767423 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 93936.3us
W20260430 07:59:15.794745 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.666776 12012 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260430 07:59:15.791656 9863 raft_consensus.cc:1240] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:15.677939 12203 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:15.796562 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 123726us
W20260430 07:59:15.798410 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.798882 9859 raft_consensus.cc:2468] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:15.804523 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:15.804855 12189 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:15.805437 12189 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:15.807541 12189 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:15.812250 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.814198 9977 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:15.816202 10008 raft_consensus.cc:1275] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260430 07:59:15.818773 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:15.820093 10004 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260430 07:59:15.822913 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.825896 10534 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:15.830096 9861 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561"
dest_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:37838
I20260430 07:59:15.830787 9861 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:59:15.831398 9861 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:15.830646 12049 raft_consensus.cc:2804] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:15.841554 12049 raft_consensus.cc:697] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:15.842257 12247 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:15.843611 12247 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.844038 12049 consensus_queue.cc:237] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:15.840881 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:15.826429 12033 consensus_meta.cc:287] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.595s user 0.017s sys 0.015s
I20260430 07:59:15.863924 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:15.866379 9903 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:15.867177 9861 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.864341 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:15.880324 9861 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:15.881469 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:15.872081 9936 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:15.856616 11995 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:15.898818 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:15.905601 9861 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:15.837787 12203 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:15.881978 12244 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:59:15.910234 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:15.912912 9931 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260430 07:59:15.913715 9931 pending_rounds.cc:85] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 4
I20260430 07:59:15.914000 9931 pending_rounds.cc:107] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260430 07:59:15.914225 9931 raft_consensus.cc:2889] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:15.910267 10568 raft_consensus.cc:1240] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260430 07:59:15.904687 9828 connection.cc:254] RPC call timeout handler was delayed by -1.8769358590000138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19530000000000003s for a response
W20260430 07:59:15.933365 9828 connection.cc:254] RPC call timeout handler was delayed by -1.8040358590000096s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2034s for a response
I20260430 07:59:15.936210 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:15.941181 9930 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
W20260430 07:59:15.942793 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 116956us
W20260430 07:59:15.944389 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 90329us
I20260430 07:59:15.929749 12241 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:15.905876 11769 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:15.955803 9857 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:15.934695 11789 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:15.976266 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 182155us
I20260430 07:59:15.972218 12242 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:15.953701 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:15.988978 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:15.962805 11995 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:15.988818 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:15.992072 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:15.973244 12196 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:15.995944 9930 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:15.996347 9857 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:15.998558 10011 raft_consensus.cc:1275] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:16.001804 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:16.002602 12251 log.cc:927] Time spent T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced: Append to log took a long time: real 0.155s user 0.002s sys 0.000s
I20260430 07:59:16.003199 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:16.003480 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:16.002637 12239 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:16.002717 10007 raft_consensus.cc:2393] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:59:16.004446 12247 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:15.978314 12189 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.027s
I20260430 07:59:16.013799 12198 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:16.015322 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:16.013999 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:16.014272 12242 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:16.012317 9977 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:15.976295 12241 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:16.017728 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:16.019047 11995 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:16.017858 12198 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:16.030296 11995 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:16.007195 12252 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:16.030725 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:16.031116 12252 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:16.032838 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:16.038945 12247 raft_consensus.cc:2749] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:16.040737 12247 consensus_queue.cc:1059] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:59:16.041074 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:16.045251 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:16.048698 10005 raft_consensus.cc:2393] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
W20260430 07:59:16.049537 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 66611.9us
I20260430 07:59:16.051112 9978 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:16.053469 9860 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:16.059152 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:16.060959 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:16.054316 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:16.043774 12252 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:16.020569 12180 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:16.143851 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:16.151054 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:16.145952 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 158187us
I20260430 07:59:16.154109 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:16.155051 9860 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260430 07:59:16.155231 10006 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:16.156594 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:16.157472 10011 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:16.162289 9934 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:16.162525 9933 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:16.148661 12242 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:16.189169 9931 raft_consensus.cc:2393] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 7: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:16.193521 9861 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
W20260430 07:59:16.196542 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 211177us
I20260430 07:59:16.143795 11841 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:16.180120 12203 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:16.245218 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:16.197571 9929 raft_consensus.cc:1275] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260430 07:59:16.404026 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 206261us
W20260430 07:59:16.235555 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 288458us
W20260430 07:59:16.439950 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:16.501056 9828 connection.cc:254] RPC call timeout handler was delayed by -1.6996358590000114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.215s for a response
W20260430 07:59:16.502871 9828 connection.cc:254] RPC call timeout handler was delayed by -1.6140364119999617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
W20260430 07:59:16.504447 9828 connection.cc:254] RPC call timeout handler was delayed by -1.5949498489999883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22400000000000003s for a response
W20260430 07:59:16.508762 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:16.510212 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:16.511809 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:16.464968 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:16.201170 9977 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:16.235692 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:16.532276 10005 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:16.533233 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:16.179889 12248 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:16.188323 12244 raft_consensus.cc:2804] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260430 07:59:16.363377 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 312219us
I20260430 07:59:16.550083 10006 raft_consensus.cc:1240] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260430 07:59:16.533398 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:16.544317 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:16.360929 12217 log.cc:927] Time spent T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20260430 07:59:16.188258 11789 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:16.558378 9930 consensus_meta.cc:287] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.542s user 0.011s sys 0.000s
I20260430 07:59:16.558825 9930 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:16.560761 9931 raft_consensus.cc:2393] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:16.233551 12249 log.cc:927] Time spent T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.270s user 0.001s sys 0.000s
I20260430 07:59:16.212111 12012 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.352s
I20260430 07:59:16.563637 12256 consensus_queue.cc:1059] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:16.425944 12239 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:16.564780 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:16.573988 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:16.574412 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:16.440048 12241 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:16.499192 12180 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:16.554489 9903 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.013s sys 0.025s
I20260430 07:59:16.573755 9860 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:16.439659 12198 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:16.595356 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 72177.2us
W20260430 07:59:16.615633 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:16.402182 12033 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:16.634479 9903 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s user 0.013s sys 0.025s
W20260430 07:59:16.669962 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 717241us
I20260430 07:59:16.664635 9861 raft_consensus.cc:1275] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260430 07:59:16.671060 9861 pending_rounds.cc:85] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 2
I20260430 07:59:16.672248 9861 pending_rounds.cc:107] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260430 07:59:16.672593 9861 raft_consensus.cc:2889] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260430 07:59:16.684399 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 504508us
I20260430 07:59:16.562772 12244 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:16.537761 11841 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:16.572407 12203 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:16.699954 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:16.659799 12248 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:16.927772 11655 consensus_meta.cc:287] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 1.304s user 0.014s sys 0.007s
I20260430 07:59:16.691126 11789 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:16.957381 12263 log.cc:927] Time spent T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
I20260430 07:59:16.957775 12248 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:16.958694 11789 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:16.958585 12203 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:16.960661 11841 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:16.787122 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 93069.1us
I20260430 07:59:16.683465 12002 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:16.836911 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:16.977205 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 292484us
I20260430 07:59:16.978209 9978 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:16.958720 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:16.927150 9977 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:16.934288 10011 consensus_meta.cc:287] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.746s user 0.012s sys 0.003s
I20260430 07:59:16.981240 10011 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:16.957368 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:16.982999 10003 raft_consensus.cc:2368] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:17.007477 10007 raft_consensus.cc:2393] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
W20260430 07:59:16.835322 12268 log.cc:927] Time spent T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
I20260430 07:59:17.016888 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.015563 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:17.015353 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:16.963768 12197 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:16.673523 12202 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.340s
I20260430 07:59:16.669726 12157 consensus_queue.cc:1048] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.346s
W20260430 07:59:17.073575 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.051335 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:17.075084 10003 raft_consensus.cc:3055] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:17.076346 10003 raft_consensus.cc:740] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:16.786634 12033 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:17.006814 12002 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:17.050334 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:16.670325 11826 consensus_queue.cc:1059] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.350s
W20260430 07:59:16.925947 12273 log.cc:927] Time spent T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.086s user 0.003s sys 0.000s
I20260430 07:59:16.961587 12203 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:16.961987 11841 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:17.039376 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 61266.5us
W20260430 07:59:17.071655 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 93051.4us
I20260430 07:59:17.099907 9904 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:17.089295 10534 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.012s sys 0.024s
I20260430 07:59:17.042068 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:17.025203 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:17.101593 10534 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.704s user 0.013s sys 0.024s
W20260430 07:59:17.102151 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:16.836126 11626 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:17.073594 9829 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s user 0.003s sys 0.007s
W20260430 07:59:17.085875 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 286526us
I20260430 07:59:17.087996 10011 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:16.987913 12244 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:17.039284 12248 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:17.062150 12196 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:17.090687 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.039873 9828 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:17.093309 12252 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:17.059885 9747 catalog_manager.cc:5201] ChangeConfig:REMOVE_PEER RPC for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 48e81934d52a4e93be5171ceea193032 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
W20260430 07:59:17.159791 10005 consensus_meta.cc:287] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.602s user 0.010s sys 0.000s
W20260430 07:59:17.135623 12220 log.cc:927] Time spent T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 1.078s user 0.017s sys 0.018s
I20260430 07:59:17.115172 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 249) took 1184 ms. Trace:
I20260430 07:59:17.107503 10003 consensus_queue.cc:260] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:17.118486 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 276) took 1176 ms. Trace:
I20260430 07:59:17.107414 10006 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:17.134752 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1027) took 1241 ms. Trace:
I20260430 07:59:17.071151 12197 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:17.147126 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:17.177834 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.147877 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.135246 10007 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:17.113981 12266 log.cc:927] Time spent T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.161s user 0.000s sys 0.001s
I20260430 07:59:17.153419 12284 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:17.136049 12033 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:17.154114 9977 leader_election.cc:400] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:17.123024 12002 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:17.133185 11769 consensus_meta.cc:287] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 1.161s user 0.010s sys 0.001s
I20260430 07:59:17.154683 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:17.155591 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.113075 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:17.158689 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.137624 9978 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:17.159945 9747 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 100268us
I20260430 07:59:17.160171 10005 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
W20260430 07:59:17.130939 12258 log.cc:927] Time spent T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.983s user 0.020s sys 0.009s
I20260430 07:59:17.111119 12283 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:17.171241 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.124303 11826 raft_consensus.cc:3055] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:17.119948 11995 raft_consensus.cc:3055] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:17.113214 12282 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:17.135278 12281 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.077s
I20260430 07:59:17.116958 12275 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.096s
I20260430 07:59:17.106648 11841 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:17.168787 9932 rpcz_store.cc:276] 0430 07:59:15.942025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:15.971989 (+ 29964us) service_pool.cc:225] Handling call
0430 07:59:15.984679 (+ 12690us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:16.008096 (+ 23417us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:16.008132 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:16.034004 (+ 25872us) log.cc:844] Serialized 26 byte log entry
0430 07:59:16.530989 (+496985us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:16.531054 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:16.531169 (+ 115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:17.118137 (+586968us) raft_consensus.cc:1685] finished
0430 07:59:17.118161 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:17.118320 (+ 159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14147,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":2012,"thread_start_us":477249,"threads_started":1,"wal-append.queue_time_us":965544}
I20260430 07:59:17.321693 9977 leader_election.cc:403] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260430 07:59:17.322741 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 185196us
I20260430 07:59:17.296231 10003 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:17.263366 10006 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:17.100896 12049 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s user 0.012s sys 0.012s
I20260430 07:59:17.329640 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.224915 9933 raft_consensus.cc:1275] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:17.212534 9935 raft_consensus.cc:1240] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:17.329226 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.264887 10004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 247) took 1380 ms. Trace:
I20260430 07:59:17.259473 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.335284 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 1059) took 1122 ms. Trace:
I20260430 07:59:17.339274 10005 rpcz_store.cc:276] 0430 07:59:16.203922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:16.204091 (+ 169us) service_pool.cc:225] Handling call
0430 07:59:17.326009 (+1121918us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:17.290786 10563 consensus_meta.cc:287] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 1.295s user 0.013s sys 0.005s
I20260430 07:59:17.348474 9860 raft_consensus.cc:1275] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:17.348999 9859 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
I20260430 07:59:17.351502 10009 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:17.239182 12285 log.cc:927] Time spent T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20260430 07:59:17.161708 12196 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:17.371436 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.325578 11826 raft_consensus.cc:740] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:17.238442 12261 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:17.168403 10008 rpcz_store.cc:276] 0430 07:59:15.931282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:15.931473 (+ 191us) service_pool.cc:225] Handling call
0430 07:59:15.957990 (+ 26517us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:16.033857 (+ 75867us) raft_consensus.cc:1534] Early marking committed up to index 6
0430 07:59:16.041867 (+ 8010us) log.cc:844] Serialized 14 byte log entry
0430 07:59:16.051868 (+ 10001us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:16.051918 (+ 50us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:16.051964 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:16.263273 (+211309us) spinlock_profiling.cc:243] Waited 3.68 ms on lock 0x7b180037cea0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661e5a33 00007fb5661e7e01 00007fb5661e7db9 00007fb5661e7d49 00007fb5661e7d11 00007fb5661e700d 0000000000379254 0000000000379099 00007fb5661c8fba
0430 07:59:17.114662 (+851389us) spinlock_profiling.cc:243] Waited 75.6 ms on lock 0x7b54002c4a54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:17.114741 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:17.115060 (+ 319us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25332,"prepare.queue_time_us":15676,"prepare.run_cpu_time_us":8911,"prepare.run_wall_time_us":97349,"raft.queue_time_us":22545,"raft.run_cpu_time_us":9135,"raft.run_wall_time_us":80905,"spinlock_wait_cycles":182377728}
W20260430 07:59:17.334013 12049 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s user 0.012s sys 0.012s
I20260430 07:59:17.169203 9937 rpcz_store.cc:276] 0430 07:59:15.833436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:15.833643 (+ 207us) service_pool.cc:225] Handling call
0430 07:59:15.858170 (+ 24527us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:15.921419 (+ 63249us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:15.921476 (+ 57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:15.923492 (+ 2016us) log.cc:844] Serialized 52 byte log entry
0430 07:59:16.040784 (+117292us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b1800482c20. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661c8587 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e5242 00007fb568e1c5ab 00007fb568e220c3 00007fb568e22471 00007fb56923edef
0430 07:59:16.193547 (+152763us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:16.435071 (+241524us) spinlock_profiling.cc:243] Waited 6.06 ms on lock 0x7b1800482c20. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661e5b62 00007fb5661e7e01 00007fb5661e7db9 00007fb5661e7d49 00007fb5661e7d11 00007fb5661e700d 0000000000379254 0000000000379099 00007fb5661c8fba
0430 07:59:16.626561 (+191490us) log.cc:844] Serialized 14 byte log entry
0430 07:59:17.072469 (+445908us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:17.072631 (+ 162us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:17.072753 (+ 122us) raft_consensus.cc:1685] finished
0430 07:59:17.072804 (+ 51us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:17.074783 (+ 1979us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56954,"prepare.queue_time_us":117977,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":17611,"raft.queue_time_us":365404,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":34406,"spinlock_wait_cycles":72283904,"thread_start_us":938257,"threads_started":3,"wal-append.queue_time_us":672710}
I20260430 07:59:17.374392 10005 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260430 07:59:17.369391 10533 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.007s sys 0.012s
I20260430 07:59:17.326448 11995 raft_consensus.cc:740] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:17.370729 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.353991 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.378834 9933 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:59:17.294790 12197 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:17.326596 12283 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:17.396375 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:17.363263 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.364168 9929 raft_consensus.cc:2376] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6. Re-sending same reply.
I20260430 07:59:17.337878 10004 rpcz_store.cc:276] 0430 07:59:15.884429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:15.887790 (+ 3361us) service_pool.cc:225] Handling call
0430 07:59:15.903833 (+ 16043us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:15.931805 (+ 27972us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:15.931858 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:15.953019 (+ 21161us) log.cc:844] Serialized 26 byte log entry
0430 07:59:16.191479 (+238460us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:16.191527 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:16.191611 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:17.264641 (+1073030us) raft_consensus.cc:1685] finished
0430 07:59:17.264671 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:17.264817 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8834,"raft.queue_time_us":26749,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":13420,"thread_start_us":203185,"threads_started":1,"wal-append.queue_time_us":691087}
I20260430 07:59:17.340847 12180 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:17.299957 12244 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:17.375394 10533 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s user 0.008s sys 0.012s
I20260430 07:59:17.423702 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.423823 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:17.347625 12260 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.882s user 0.010s sys 0.041s
I20260430 07:59:17.383673 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:17.335638 11769 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:17.388641 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.166153 12248 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:17.393043 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 239045us
I20260430 07:59:17.393523 12256 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:17.366667 10563 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:17.212893 12290 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:17.437800 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.369840 12202 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:17.430694 12260 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.992s user 0.010s sys 0.041s
I20260430 07:59:17.374675 12196 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:17.427067 9938 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:17.428423 10003 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:17.427896 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.400481 12180 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:17.400419 12283 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:17.374884 12282 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:17.384545 12284 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:17.381505 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:17.421507 9932 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
I20260430 07:59:17.439949 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.603942 11769 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:17.556133 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1608) took 1011 ms. Trace:
I20260430 07:59:17.466356 12290 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:17.640909 12202 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:17.526494 9902 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.045s sys 0.119s
I20260430 07:59:17.649574 9934 raft_consensus.cc:2368] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7.
I20260430 07:59:17.526134 11841 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:17.650864 9902 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.843s user 0.068s sys 0.142s
I20260430 07:59:17.644069 11769 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:17.608677 11995 consensus_queue.cc:260] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:17.535830 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:17.514637 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.662370 12202 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:17.659554 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.431829 12197 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.234s
I20260430 07:59:17.667744 9930 raft_consensus.cc:1240] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:17.665361 11769 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:17.648404 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.670392 12282 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:17.438841 12256 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:17.674510 12246 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260430 07:59:17.676347 12180 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:17.684501 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 663461us
I20260430 07:59:17.686991 9828 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:17.678334 12196 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:17.466599 10563 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54778 (request call id 125) took 1556 ms (client timeout 1871 ms). Trace:
I20260430 07:59:17.551694 9977 leader_election.cc:400] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:17.470813 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.691604 9937 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:17.683343 9932 raft_consensus.cc:2393] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
W20260430 07:59:17.690631 10563 rpcz_store.cc:269] 0430 07:59:15.884751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:15.890414 (+ 5663us) service_pool.cc:225] Handling call
0430 07:59:17.440613 (+1550199us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:17.681697 9861 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.003s sys 0.009s
I20260430 07:59:17.376503 10005 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260430 07:59:17.694720 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.642630 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:17.698565 9861 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s user 0.003s sys 0.009s
I20260430 07:59:17.659009 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:17.703110 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.690868 9977 leader_election.cc:403] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20260430 07:59:17.703550 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.701807 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:17.702239 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.702260 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 275) took 1799 ms. Trace:
I20260430 07:59:17.440378 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.708155 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:17.711274 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.401247 12261 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:17.705477 9936 rpcz_store.cc:276] 0430 07:59:15.902901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:15.903104 (+ 203us) service_pool.cc:225] Handling call
0430 07:59:15.934674 (+ 31570us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:16.388558 (+453884us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:16.388608 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:16.455491 (+ 66883us) log.cc:844] Serialized 26 byte log entry
0430 07:59:16.975522 (+520031us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:16.975591 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:16.975712 (+ 121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:17.701901 (+726189us) raft_consensus.cc:1685] finished
0430 07:59:17.701926 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:17.702086 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2743,"raft.queue_time_us":28481,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":6522,"thread_start_us":708223,"threads_started":2,"wal-append.queue_time_us":591689}
I20260430 07:59:17.656363 11995 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:17.694669 12256 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:17.449887 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:17.716289 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:17.717476 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 257224us
I20260430 07:59:17.656419 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:17.722697 9860 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:17.723376 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.699178 12284 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:17.724154 9932 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:17.719337 9931 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:17.720433 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.727674 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.728999 9933 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20260430 07:59:17.738124 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.729628 9933 pending_rounds.cc:85] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 2
I20260430 07:59:17.768703 9933 pending_rounds.cc:107] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260430 07:59:17.769480 9933 raft_consensus.cc:2889] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:17.779815 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.786756 9931 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260430 07:59:17.736945 12239 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:17.789228 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:59:17.708443 12297 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:17.717893 12252 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:17.790017 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 103288us
I20260430 07:59:17.779801 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.791304 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:17.796833 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.801028 9857 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 8
I20260430 07:59:17.793547 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:17.800455 9858 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:17.798404 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:17.803858 12292 log.cc:927] Time spent T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260430 07:59:17.804265 9862 raft_consensus.cc:2410] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260430 07:59:17.805413 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.643105 9863 rpcz_store.cc:276] 0430 07:59:16.544364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:16.544587 (+ 223us) service_pool.cc:225] Handling call
0430 07:59:16.572608 (+ 28021us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:16.572933 (+ 325us) raft_consensus.cc:1534] Early marking committed up to index 4
0430 07:59:16.971238 (+398305us) log.cc:844] Serialized 14 byte log entry
0430 07:59:17.551957 (+580719us) log.cc:844] Serialized 14 byte log entry
0430 07:59:17.555614 (+ 3657us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:17.555692 (+ 78us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:17.555757 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:17.555860 (+ 103us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:17.556053 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16937,"prepare.queue_time_us":567454,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":128,"thread_start_us":828438,"threads_started":3,"wal-append.queue_time_us":416362}
I20260430 07:59:17.806207 12290 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:17.806823 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.803756 9936 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:17.807351 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1064) took 1061 ms. Trace:
I20260430 07:59:17.803668 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.808844 10010 rpcz_store.cc:276] 0430 07:59:16.746389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:17.077288 (+330899us) service_pool.cc:225] Handling call
0430 07:59:17.085459 (+ 8171us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:17.123873 (+ 38414us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:17.123906 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:17.124402 (+ 496us) log.cc:844] Serialized 26 byte log entry
0430 07:59:17.431063 (+306661us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:17.431095 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:17.431150 (+ 55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:17.807073 (+375923us) raft_consensus.cc:1685] finished
0430 07:59:17.807098 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:17.807277 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34134,"raft.queue_time_us":34479,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":93,"thread_start_us":306277,"threads_started":1,"wal-append.queue_time_us":679853}
I20260430 07:59:17.810590 9858 raft_consensus.cc:2393] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:17.813544 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1065) took 1066 ms. Trace:
I20260430 07:59:17.813822 9828 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:17.817158 9932 raft_consensus.cc:2368] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:17.802830 9935 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:17.813992 10011 rpcz_store.cc:276] 0430 07:59:16.746841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:17.162901 (+416060us) service_pool.cc:225] Handling call
0430 07:59:17.263318 (+100417us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:17.263602 (+ 284us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:17.263628 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:17.264171 (+ 543us) log.cc:844] Serialized 26 byte log entry
0430 07:59:17.714093 (+449922us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:17.714260 (+ 167us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:17.714531 (+ 271us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:17.813341 (+ 98810us) raft_consensus.cc:1685] finished
0430 07:59:17.813363 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:17.813485 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":449207,"threads_started":1,"wal-append.queue_time_us":547330}
W20260430 07:59:17.824837 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 102523us
I20260430 07:59:17.715065 12300 raft_consensus.cc:2764] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260430 07:59:17.830927 9904 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:17.837378 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.857201 9938 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:17.790750 12248 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:17.803715 11841 raft_consensus.cc:2764] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260430 07:59:17.871709 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 58034.2us
I20260430 07:59:17.870797 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:17.872428 9860 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:17.737787 12235 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.122s
W20260430 07:59:17.891680 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:17.892638 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.834992 12261 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:17.919163 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.916038 9929 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260430 07:59:17.916831 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:17.889510 11769 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:17.948053 9863 raft_consensus.cc:1240] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: []
W20260430 07:59:17.917964 12294 log.cc:927] Time spent T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.181s user 0.000s sys 0.001s
W20260430 07:59:17.943900 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.950860 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:17.951864 9860 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
W20260430 07:59:17.952500 9977 connection.cc:254] RPC call timeout handler was delayed by -2.5266643640000552s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
W20260430 07:59:17.953089 9977 connection.cc:254] RPC call timeout handler was delayed by -2.4528643640000496s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
W20260430 07:59:17.953506 9977 connection.cc:254] RPC call timeout handler was delayed by -2.4465643640000394s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1644s for a response
W20260430 07:59:17.953984 9977 connection.cc:254] RPC call timeout handler was delayed by -2.3682643640000265s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17310000000000003s for a response
W20260430 07:59:17.954385 9977 connection.cc:254] RPC call timeout handler was delayed by -2.3232643640000674s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1781s for a response
W20260430 07:59:17.954763 9977 connection.cc:254] RPC call timeout handler was delayed by -2.2530643640000676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
W20260430 07:59:17.955845 9977 connection.cc:254] RPC call timeout handler was delayed by -2.2485643640000603s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
I20260430 07:59:17.904690 12196 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:17.886947 12244 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:17.891911 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:17.963850 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:17.939757 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:17.964484 9860 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:17.965626 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:17.961395 9977 connection.cc:254] RPC call timeout handler was delayed by -2.169364364000046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
I20260430 07:59:17.961714 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:17.969908 9977 connection.cc:254] RPC call timeout handler was delayed by -2.1486643640000693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1975s for a response
I20260430 07:59:17.969708 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:17.970541 9977 connection.cc:254] RPC call timeout handler was delayed by -2.134264364000046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19910000000000003s for a response
I20260430 07:59:17.916220 12303 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:59:17.971050 9977 connection.cc:254] RPC call timeout handler was delayed by -2.043657579000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
W20260430 07:59:17.978216 9977 connection.cc:254] RPC call timeout handler was delayed by -1.992964364000045s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21480000000000002s for a response
W20260430 07:59:17.978693 9977 connection.cc:254] RPC call timeout handler was delayed by -1.981264364000026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21610000000000001s for a response
W20260430 07:59:17.979194 9977 connection.cc:254] RPC call timeout handler was delayed by -1.9500575790000312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1761s for a response
W20260430 07:59:17.977712 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 179326us
W20260430 07:59:17.945050 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:17.979904 9977 connection.cc:254] RPC call timeout handler was delayed by -1.904764364000016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22460000000000002s for a response
W20260430 07:59:17.976480 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:17.979245 9860 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:17.983372 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 183761us
I20260430 07:59:17.982621 9978 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:17.947496 11769 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:17.990497 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 62640.4us
I20260430 07:59:17.971056 12246 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:17.829671 12297 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:17.983421 12281 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:18.016995 9978 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:18.018415 11626 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:18.020272 9828 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:17.992751 12244 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:17.988294 12303 raft_consensus.cc:697] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:17.861996 12290 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.067s
I20260430 07:59:18.025309 9938 raft_consensus.cc:3055] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:17.993777 11769 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:17.853724 12284 consensus_queue.cc:1048] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.049s
I20260430 07:59:18.038564 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1062) took 1490 ms. Trace:
I20260430 07:59:18.039062 10012 rpcz_store.cc:276] 0430 07:59:16.547954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:16.548146 (+ 192us) service_pool.cc:225] Handling call
0430 07:59:16.548655 (+ 509us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:17.007319 (+458664us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:17.007353 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:17.025092 (+ 17739us) log.cc:844] Serialized 26 byte log entry
0430 07:59:17.369850 (+344758us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:17.369896 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:17.369971 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:18.038362 (+668391us) raft_consensus.cc:1685] finished
0430 07:59:18.038385 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:18.038507 (+ 122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":351230,"raft.queue_time_us":313276,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":7793,"thread_start_us":268505,"threads_started":1,"wal-append.queue_time_us":474474}
I20260430 07:59:18.045149 12296 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:18.045879 12296 raft_consensus.cc:697] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:18.046816 12296 consensus_queue.cc:237] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.050038 9931 raft_consensus.cc:1275] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260430 07:59:18.033744 9932 raft_consensus.cc:1240] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260430 07:59:18.028810 12244 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:18.033380 9938 raft_consensus.cc:740] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:18.061582 9936 raft_consensus.cc:1240] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:18.061848 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:18.062520 9937 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:18.064414 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.065187 9930 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:18.054430 12244 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:18.006536 12283 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.053246 9938 consensus_queue.cc:260] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:18.032147 11841 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:18.078804 9935 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
W20260430 07:59:18.081506 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.083735 10012 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:18.084544 10012 raft_consensus.cc:606] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:18.084997 10012 raft_consensus.cc:614] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:18.088989 9938 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
W20260430 07:59:18.089411 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 93721.5us
I20260430 07:59:18.022365 12246 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.100063 9828 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:18.080019 11841 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:18.081959 12290 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:18.105808 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:18.107887 9935 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:18.025980 12281 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:18.125182 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 132417us
I20260430 07:59:18.129596 10533 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:18.139358 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:18.140110 10012 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:18.115830 11841 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:18.050818 11826 consensus_queue.cc:260] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.158504 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.161693 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.162789 9930 raft_consensus.cc:2410] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260430 07:59:18.166396 10009 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:18.167276 10009 raft_consensus.cc:606] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:18.167753 10009 raft_consensus.cc:614] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:18.171689 9828 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:18.185932 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.192224 9936 raft_consensus.cc:3055] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:18.192704 9936 raft_consensus.cc:740] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:18.194715 9934 raft_consensus.cc:1240] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:18.197695 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:18.198354 9930 raft_consensus.cc:2376] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5. Re-sending same reply.
W20260430 07:59:18.208611 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.134745 12290 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.214402 9931 raft_consensus.cc:1275] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:18.220027 12283 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:18.223284 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:18.224296 9937 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:18.226718 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.202814 11826 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:18.234015 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 69692.5us
W20260430 07:59:18.243235 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.253911 10009 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:18.226572 12283 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:18.270747 9936 consensus_queue.cc:260] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:18.292568 9828 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:18.294255 9938 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260430 07:59:18.292644 10009 raft_consensus.cc:606] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
W20260430 07:59:18.295749 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 280532us
W20260430 07:59:18.296211 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.209865 12235 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
W20260430 07:59:18.305394 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 280247us
I20260430 07:59:18.312537 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:18.298424 11995 consensus_meta.cc:287] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.574s user 0.010s sys 0.007s
I20260430 07:59:18.206646 12303 consensus_queue.cc:237] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.334313 9938 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 3.
W20260430 07:59:18.318471 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:18.346184 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:18.334299 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:18.336467 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.294700 12300 raft_consensus.cc:2764] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260430 07:59:18.342604 9936 raft_consensus.cc:3060] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:18.303866 12180 consensus_meta.cc:287] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.613s user 0.009s sys 0.005s
W20260430 07:59:18.347620 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:18.348006 9857 consensus_meta.cc:287] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.517s user 0.011s sys 0.000s
I20260430 07:59:18.313015 12297 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:18.301028 12302 raft_consensus.cc:2764] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260430 07:59:18.370343 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:18.373056 10009 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:18.373374 9938 raft_consensus.cc:2376] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7. Re-sending same reply.
I20260430 07:59:18.357231 9828 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:18.341579 11626 raft_consensus.cc:697] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:18.310539 11769 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:18.352299 12305 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.274s user 0.001s sys 0.001s
I20260430 07:59:18.296422 12307 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:18.368309 9857 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:18.305698 12283 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.325208 9935 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:18.368613 12311 log.cc:927] Time spent T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.148s user 0.000s sys 0.002s
I20260430 07:59:18.380306 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:18.387401 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 369370us
I20260430 07:59:18.393347 9933 raft_consensus.cc:1240] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:18.395321 9828 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:18.380031 12301 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:18.399537 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.400368 9978 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:18.367029 12180 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.387241 12246 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:18.392426 12302 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:18.410770 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.423228 9935 raft_consensus.cc:2376] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7. Re-sending same reply.
W20260430 07:59:18.412281 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:18.424659 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:18.426887 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.434787 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.436285 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.438426 9931 raft_consensus.cc:1275] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:18.438544 12283 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:18.439692 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:18.441208 12310 log.cc:927] Time spent T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260430 07:59:18.440534 9931 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:18.441648 11626 consensus_queue.cc:237] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.442579 9828 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:18.443073 9977 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:18.443276 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.439765 12281 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:18.441980 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.443899 9828 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:18.448107 9930 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:18.448402 9977 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:18.444608 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.449857 9937 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
W20260430 07:59:18.445886 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 95610.2us
I20260430 07:59:18.446378 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:18.446738 9978 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:18.446945 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1067) took 1699 ms. Trace:
I20260430 07:59:18.450737 9857 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:18.452657 9904 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:18.452909 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.453840 9935 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:18.378021 10004 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.001s sys 0.023s
W20260430 07:59:18.456486 10004 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s user 0.002s sys 0.023s
I20260430 07:59:18.450769 10007 rpcz_store.cc:276] 0430 07:59:16.747514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:17.216354 (+468840us) service_pool.cc:225] Handling call
0430 07:59:17.271998 (+ 55644us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:17.349987 (+ 77989us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:17.350033 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:17.362309 (+ 12276us) log.cc:844] Serialized 26 byte log entry
0430 07:59:17.716067 (+353758us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:17.716107 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:17.716183 (+ 76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:18.445242 (+729059us) raft_consensus.cc:1685] finished
0430 07:59:18.445284 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:18.446890 (+ 1606us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9719,"raft.queue_time_us":34510,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":19262,"thread_start_us":289274,"threads_started":1,"wal-append.queue_time_us":624224}
I20260430 07:59:18.460909 9978 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:18.461974 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.461926 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.462841 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:18.462875 9938 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:18.410480 12296 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:18.464646 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.465317 10533 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:18.407547 12290 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:18.466902 9930 raft_consensus.cc:1275] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:18.422295 12203 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:18.444629 12302 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:18.475996 10006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1068) took 1728 ms. Trace:
W20260430 07:59:18.477174 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.479317 12290 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:18.482924 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.461159 12283 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:18.456825 12297 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:18.516268 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.539341 9929 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:18.466943 12196 raft_consensus.cc:2804] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:18.506557 10006 rpcz_store.cc:276] 0430 07:59:16.747871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:17.352876 (+605005us) service_pool.cc:225] Handling call
0430 07:59:17.353321 (+ 445us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:17.402285 (+ 48964us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:17.402333 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:17.719932 (+317599us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b2c0008b928. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb567b7dc81 00007fb567b8d5e2 00007fb5692bede1 00007fb56928381a 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22
0430 07:59:17.737192 (+ 17260us) log.cc:844] Serialized 26 byte log entry
0430 07:59:18.141543 (+404351us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:18.141587 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:18.141664 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:18.468601 (+326937us) raft_consensus.cc:1685] finished
0430 07:59:18.468628 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:18.475883 (+ 7255us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1549,"raft.queue_time_us":21282,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":13179,"spinlock_wait_cycles":24788864,"thread_start_us":333092,"threads_started":1,"wal-append.queue_time_us":426781}
I20260430 07:59:18.466596 12248 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:18.452504 12246 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:18.513726 12242 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:59:18.632184 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.549945 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:18.492094 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.467814 12281 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:18.473135 12296 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:18.615124 12328 log.cc:927] Time spent T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.212s user 0.000s sys 0.001s
I20260430 07:59:18.622506 12327 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:18.617484 12252 consensus_meta.cc:287] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.795s user 0.013s sys 0.001s
I20260430 07:59:18.651662 12252 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:18.652568 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:18.653251 9929 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:18.657400 12296 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:18.661732 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:18.539398 12256 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:18.593034 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:18.538156 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.633111 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:18.623384 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 158122us
I20260430 07:59:18.637429 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:18.638001 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.637385 9857 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
W20260430 07:59:18.575883 12309 log.cc:927] Time spent T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.274s user 0.000s sys 0.001s
I20260430 07:59:18.475023 12203 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:18.646507 11769 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:18.644600 12242 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.683357 12242 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:18.616612 12244 consensus_meta.cc:287] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.549s user 0.012s sys 0.000s
I20260430 07:59:18.676533 12203 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:18.684094 12244 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.686188 10004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 255) took 1250 ms. Trace:
I20260430 07:59:18.686602 10004 rpcz_store.cc:276] 0430 07:59:17.436086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:17.438104 (+ 2018us) service_pool.cc:225] Handling call
0430 07:59:17.623665 (+185561us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:17.705817 (+ 82152us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:17.705853 (+ 36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:17.768723 (+ 62870us) log.cc:844] Serialized 26 byte log entry
0430 07:59:18.473666 (+704943us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:18.473729 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:18.473860 (+ 131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:18.685986 (+212126us) raft_consensus.cc:1685] finished
0430 07:59:18.686005 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:18.686131 (+ 126us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":81669,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":14051,"thread_start_us":621590,"threads_started":1,"wal-append.queue_time_us":682786}
I20260430 07:59:18.687693 12244 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:18.688462 10009 raft_consensus.cc:606] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:18.688884 10009 raft_consensus.cc:614] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:18.690006 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.690908 9933 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:18.693442 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.694100 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:18.694752 9933 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:18.695082 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.672681 9930 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:18.673556 12314 log.cc:927] Time spent T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.296s user 0.001s sys 0.000s
I20260430 07:59:18.700341 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:18.706461 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:18.710709 9933 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:18.515987 12302 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:18.531782 12283 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:18.665920 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 222032us
I20260430 07:59:18.666395 9936 raft_consensus.cc:1275] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260430 07:59:18.716643 9936 pending_rounds.cc:85] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 5
I20260430 07:59:18.716890 9936 pending_rounds.cc:107] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260430 07:59:18.717108 9936 raft_consensus.cc:2889] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260430 07:59:18.630432 12329 log.cc:927] Time spent T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260430 07:59:18.538908 12297 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:18.533025 12290 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.679109 9929 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:18.679883 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 372) took 1533 ms. Trace:
I20260430 07:59:18.680104 12180 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:18.676707 12196 raft_consensus.cc:697] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:59:18.590536 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.711992 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.727555 9938 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:18.712374 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:18.674131 12281 raft_consensus.cc:697] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:18.714314 11841 consensus_meta.cc:287] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.529s user 0.003s sys 0.008s
I20260430 07:59:18.721518 9861 rpcz_store.cc:276] 0430 07:59:17.146812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:17.146964 (+ 152us) service_pool.cc:225] Handling call
0430 07:59:17.147402 (+ 438us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:17.708910 (+561508us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:17.708948 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:17.722922 (+ 13974us) log.cc:844] Serialized 26 byte log entry
0430 07:59:17.982842 (+259920us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:17.982884 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:17.982965 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:18.679632 (+696667us) raft_consensus.cc:1685] finished
0430 07:59:18.679657 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:18.679800 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":293219,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":79611,"thread_start_us":733337,"threads_started":2,"wal-append.queue_time_us":406872}
I20260430 07:59:18.722020 9978 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:18.728847 9857 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:18.733806 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.713644 12256 raft_consensus.cc:697] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 8 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:59:18.735359 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 142478us
I20260430 07:59:18.735477 9828 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:18.720007 12283 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:18.724282 12297 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:18.739744 12303 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:18.740242 12303 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:18.739920 12281 consensus_queue.cc:237] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.740655 12303 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:18.741217 12290 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:18.741357 12337 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:18.743360 12337 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:18.743721 12337 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:18.749922 9828 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:18.750610 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 77237.7us
I20260430 07:59:18.736285 11841 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.756542 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:18.757489 9929 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:18.758791 9863 raft_consensus.cc:1240] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:18.728794 12336 consensus_queue.cc:1048] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.055s
I20260430 07:59:18.758415 12196 consensus_queue.cc:237] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:18.759644 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.038595 9857 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
W20260430 07:59:18.925410 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:18.862349 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.858685 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:18.790697 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:18.905380 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:18.798650 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 46386.8us
W20260430 07:59:18.858633 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:18.917567 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.043650 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.044201 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.044574 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.044979 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.045616 9929 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
W20260430 07:59:19.042021 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:18.816579 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.046684 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.046980 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:18.904680 12239 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:19.048100 12239 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:19.050770 12239 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:19.051931 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:19.055034 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.044379 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.042418 9934 raft_consensus.cc:2376] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4. Re-sending same reply.
I20260430 07:59:19.057222 9930 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:19.063212 9828 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:19.087868 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:19.074980 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.090986 9933 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:19.088688 9938 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:19.094787 9828 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:19.069713 10010 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:19.094213 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.103490 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:18.816817 12049 consensus_queue.cc:1048] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.289s
I20260430 07:59:18.783913 12290 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:19.107527 12290 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:19.107700 9978 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:19.108013 12290 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:19.096606 10010 raft_consensus.cc:606] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:19.109460 11826 consensus_queue.cc:1059] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:59:19.109759 12340 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:18.860905 12256 consensus_queue.cc:237] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.110530 12340 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:19.065492 12252 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:19.112301 12252 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:19.114128 12180 raft_consensus.cc:3055] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:19.114432 12252 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:19.114594 12180 raft_consensus.cc:740] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:19.115619 12180 consensus_queue.cc:260] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.116745 12340 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:19.118077 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260430 07:59:19.118580 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:19.119656 12298 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:19.120136 12298 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:19.120260 12302 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.120713 12298 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:19.110524 11826 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:19.122066 11826 raft_consensus.cc:697] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:19.123015 11826 consensus_queue.cc:237] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.095897 12242 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:19.126361 12242 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:19.116918 12180 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:19.127905 9855 raft_consensus.cc:1240] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:19.087466 11626 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:19.129807 11626 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.131896 11626 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:19.132058 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:19.033679 12244 raft_consensus.cc:993] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: : Instructing follower 27f97a330f374849be58a168318c1ec6 to start an election
W20260430 07:59:19.146457 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.151523 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:19.152298 9934 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:19.004956 12203 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.087644 12338 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260430 07:59:18.998200 12297 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.098722 12327 consensus_queue.cc:1048] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.171s
W20260430 07:59:19.213836 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.221788 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 55493.3us
I20260430 07:59:19.243062 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:19.191412 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.312049 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 217270us
I20260430 07:59:19.169118 9863 raft_consensus.cc:1275] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260430 07:59:19.157449 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:19.227957 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 85256.4us
I20260430 07:59:19.272537 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:59:19.386039 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:19.278355 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:19.224728 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.152462 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:19.066409 12248 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:19.379709 9828 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:19.379542 9977 leader_election.cc:336] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.510s (SENT)
W20260430 07:59:19.417788 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.419494 9929 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:19.425329 9859 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:19.134855 12242 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:19.430735 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:19.438534 9930 raft_consensus.cc:2376] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5. Re-sending same reply.
I20260430 07:59:19.431469 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:19.434391 10534 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:59:19.348857 12283 consensus_meta.cc:287] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.595s user 0.010s sys 0.001s
I20260430 07:59:19.438131 9931 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:19.443444 9858 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:19.187667 12244 raft_consensus.cc:1081] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Signalling peer 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:59:19.441529 9863 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:19.453529 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:19.187969 11841 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:19.494586 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.456285 9934 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:19.460328 12336 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:19.495604 12336 raft_consensus.cc:697] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:19.416340 12300 raft_consensus.cc:2764] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260430 07:59:19.461570 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 81638.7us
I20260430 07:59:19.498662 9828 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:19.462240 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 82542.6us
I20260430 07:59:19.445259 12248 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:19.463776 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.467952 12296 consensus_meta.cc:287] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.810s user 0.005s sys 0.005s
W20260430 07:59:19.470721 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.471648 12239 consensus_queue.cc:1059] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260430 07:59:19.478470 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.517160 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.518074 9934 raft_consensus.cc:2368] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:19.478806 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.479580 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.480237 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.519893 9935 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:19.481084 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.520819 9828 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:19.521785 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.273591 12347 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:19.501955 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 56858.3us
I20260430 07:59:19.528110 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:19.528846 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.529309 9938 raft_consensus.cc:2393] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:19.530453 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.462411 12284 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:19.482586 12308 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:19.450259 12283 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.527716 12347 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:19.511057 12303 consensus_meta.cc:287] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.770s user 0.012s sys 0.000s
W20260430 07:59:19.514716 12337 consensus_meta.cc:287] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.769s user 0.010s sys 0.004s
I20260430 07:59:19.541693 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:19.545235 10007 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:19.510135 12281 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
W20260430 07:59:19.586988 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.587713 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 151507us
I20260430 07:59:19.533640 12203 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:19.589435 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 65903.6us
I20260430 07:59:19.607435 9932 raft_consensus.cc:2368] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7.
W20260430 07:59:19.607754 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.608249 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.541311 11626 raft_consensus.cc:3055] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Stepping down as leader of term 3
W20260430 07:59:19.600817 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 80159.5us
W20260430 07:59:19.613816 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.540022 12308 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:19.546466 12302 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:19.613194 9978 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:19.516762 11841 consensus_queue.cc:1048] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20260430 07:59:19.617504 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.627359 9978 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:19.603201 9929 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
W20260430 07:59:19.623445 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.544117 12347 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:19.532861 12296 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.632592 9978 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:19.633419 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.540903 12284 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.496909 12336 consensus_queue.cc:237] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.648573 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:19.650138 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:19.650380 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 149754us
I20260430 07:59:19.651027 9929 raft_consensus.cc:2376] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5. Re-sending same reply.
I20260430 07:59:19.651238 9977 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:19.562495 12303 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.613341 11626 raft_consensus.cc:740] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:19.626924 12308 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:19.658851 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.588361 12337 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.638284 12203 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:19.673297 9856 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:19.613629 11826 consensus_queue.cc:1048] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20260430 07:59:19.647586 12242 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:19.643224 12244 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:19.684321 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.686235 9929 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:19.687355 9860 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:19.695676 9859 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e"
dest_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:37838
I20260430 07:59:19.696609 9859 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:59:19.697121 9859 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:19.685297 12242 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:19.698830 12297 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:19.721033 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:19.727660 9929 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:19.690076 12203 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:19.699465 12244 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:19.737061 11626 consensus_queue.cc:260] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.741425 11626 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:19.733372 12242 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
W20260430 07:59:19.754932 12330 log.cc:927] Time spent T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
W20260430 07:59:19.766090 12346 log.cc:927] Time spent T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.326s user 0.001s sys 0.000s
W20260430 07:59:19.757521 12350 log.cc:927] Time spent T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.294s user 0.000s sys 0.001s
W20260430 07:59:19.757691 12353 log.cc:927] Time spent T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
W20260430 07:59:19.770154 12344 log.cc:927] Time spent T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.223s user 0.001s sys 0.000s
I20260430 07:59:19.738940 12203 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.802803 12203 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:19.795679 9859 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:19.808168 12365 log.cc:927] Time spent T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
W20260430 07:59:19.818066 12363 log.cc:927] Time spent T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20260430 07:59:19.803845 12248 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:19.823081 12298 consensus_meta.cc:287] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.702s user 0.010s sys 0.000s
W20260430 07:59:19.825245 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.830667 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.833448 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.835947 12248 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:19.840646 12248 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:19.842449 12367 log.cc:927] Time spent T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.159s user 0.002s sys 0.000s
I20260430 07:59:19.843580 12248 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:19.846222 12337 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:19.848757 10563 raft_consensus.cc:1240] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:19.849241 10565 raft_consensus.cc:1240] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:19.833956 12246 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:19.853826 12246 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:19.856029 12246 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:19.857611 12303 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:19.858657 12284 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:19.860321 12296 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:19.861209 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1460) took 1167 ms. Trace:
I20260430 07:59:19.861599 9932 rpcz_store.cc:276] 0430 07:59:18.693351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:18.694814 (+ 1463us) service_pool.cc:225] Handling call
0430 07:59:19.860975 (+1166161us) spinlock_profiling.cc:243] Waited 911 ms on lock 0x7b54001b26d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:19.861111 (+ 136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2095824640}
I20260430 07:59:19.862529 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:19.863219 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:19.864089 10007 raft_consensus.cc:2410] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:19.864362 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:19.865305 10006 raft_consensus.cc:2393] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 7: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
W20260430 07:59:19.833864 12362 log.cc:927] Time spent T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.224s user 0.001s sys 0.000s
I20260430 07:59:19.866451 10010 raft_consensus.cc:1275] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260430 07:59:19.871712 12366 log.cc:927] Time spent T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20260430 07:59:19.864293 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:19.834723 12283 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:19.879658 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:19.906838 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.835155 9859 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:19.932531 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:19.933776 9859 raft_consensus.cc:2410] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260430 07:59:19.947198 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 112106us
W20260430 07:59:19.962368 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:19.994159 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:19.823567 12298 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:19.887537 12290 consensus_meta.cc:287] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.779s user 0.009s sys 0.004s
I20260430 07:59:19.996289 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.005613 10008 raft_consensus.cc:2368] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
W20260430 07:59:20.006451 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 355173us
W20260430 07:59:20.010462 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.016109 10568 raft_consensus.cc:1240] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 8. Ops: []
I20260430 07:59:20.017228 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.005990 12360 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:19.886134 12308 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:20.037612 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.042575 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.007951 12290 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.013973 12242 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:20.050925 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.180737 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.266059 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.266228 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.219542 12355 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
W20260430 07:59:20.266909 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.200577 12281 raft_consensus.cc:697] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:20.267500 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.267387 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:20.268518 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.268913 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.269243 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:20.230393 12347 consensus_meta.cc:287] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.577s user 0.008s sys 0.004s
I20260430 07:59:20.269860 10006 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:20.269784 10011 raft_consensus.cc:2410] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:20.270107 12347 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.271329 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.268177 12281 consensus_queue.cc:237] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.272222 10011 raft_consensus.cc:2368] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:20.271325 9862 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:20.273389 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:20.134155 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.045471 12360 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:20.266127 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.274691 12347 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:20.277652 12360 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:20.146276 12298 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:20.025769 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:20.158160 10565 raft_consensus.cc:1240] T e537f351c19b43fd8b0b0a89cf897487 P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:20.281836 9861 raft_consensus.cc:1240] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:20.283043 12290 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:20.287300 12242 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:20.052843 10008 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:20.095197 12180 consensus_meta.cc:287] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.967s user 0.011s sys 0.002s
I20260430 07:59:20.304600 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:20.310781 12382 log.cc:927] Time spent T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260430 07:59:20.316984 12308 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:20.325207 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 53836.8us
I20260430 07:59:20.346079 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 389) took 1510 ms. Trace:
I20260430 07:59:20.348539 9854 rpcz_store.cc:276] 0430 07:59:18.835464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:19.061153 (+225689us) service_pool.cc:225] Handling call
0430 07:59:19.094449 (+ 33296us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:19.094638 (+ 189us) raft_consensus.cc:1534] Early marking committed up to index 6
0430 07:59:19.452487 (+357849us) log.cc:844] Serialized 14 byte log entry
0430 07:59:19.841903 (+389416us) spinlock_profiling.cc:243] Waited 7.19 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:20.345551 (+503648us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:20.345634 (+ 83us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:20.345720 (+ 86us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:20.345827 (+ 107us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:20.345994 (+ 167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":684865,"prepare.queue_time_us":93292,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":2906,"raft.queue_time_us":657388,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":75204,"spinlock_wait_cycles":16541440,"thread_start_us":557630,"threads_started":3,"wal-append.queue_time_us":191667}
W20260430 07:59:20.366852 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.367913 12203 consensus_queue.cc:1059] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:20.369918 10565 raft_consensus.cc:1240] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:20.370218 12378 raft_consensus.cc:3055] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:20.370586 12378 raft_consensus.cc:740] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:20.371217 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:20.371522 12378 consensus_queue.cc:260] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.373976 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 393) took 1241 ms. Trace:
I20260430 07:59:20.378058 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:20.374505 9857 rpcz_store.cc:276] 0430 07:59:19.133014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:19.133666 (+ 652us) service_pool.cc:225] Handling call
0430 07:59:19.153464 (+ 19798us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:19.153737 (+ 273us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:19.833764 (+680027us) spinlock_profiling.cc:243] Waited 84.6 ms on lock 0x7b1800482140. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb568e2a85c 00007fb5661c8c0b 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e2a95 00007fb56cc86837 00007fb56928dd87 00007fb56929d49f
0430 07:59:19.835473 (+ 1709us) log.cc:844] Serialized 14 byte log entry
0430 07:59:20.373572 (+538099us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:20.373641 (+ 69us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:20.373688 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:20.373761 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:20.373907 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":316599,"prepare.queue_time_us":119201,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"spinlock_wait_cycles":194659072,"thread_start_us":337002,"threads_started":2}
I20260430 07:59:20.373070 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:20.386708 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:20.393446 9859 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:20.385710 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:20.390712 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 767178us
I20260430 07:59:20.390924 10011 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:20.394425 9863 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:20.392643 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:20.396333 9978 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:20.392812 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1647) took 1254 ms. Trace:
I20260430 07:59:20.376672 12355 raft_consensus.cc:697] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:20.397493 9860 rpcz_store.cc:276] 0430 07:59:19.138240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:19.138442 (+ 202us) service_pool.cc:225] Handling call
0430 07:59:19.138656 (+ 214us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:19.687783 (+549127us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:20.392509 (+704726us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b540007fb54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:20.392733 (+ 224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1261147520}
I20260430 07:59:20.398425 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:20.398869 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:20.398654 12355 consensus_queue.cc:237] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.399461 9860 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:20.406430 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:20.416492 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:20.418040 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.417954 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:20.430960 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.431715 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.432512 10011 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:20.433905 9930 raft_consensus.cc:1275] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:20.422294 11626 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:20.436002 11626 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:20.436358 11626 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:20.437355 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.440793 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.416812 12296 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:20.442909 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:20.443601 9936 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:20.442730 12296 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.445132 9860 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
W20260430 07:59:20.446887 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.445220 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.446023 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.445641 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.450387 10005 raft_consensus.cc:2410] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260430 07:59:20.450220 10007 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:20.451046 10011 raft_consensus.cc:2368] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9.
I20260430 07:59:20.451714 10008 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:20.451973 10005 raft_consensus.cc:1240] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:20.452147 10007 raft_consensus.cc:1275] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260430 07:59:20.452072 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:20.452963 9860 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:20.445549 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:20.446694 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.453639 9978 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:20.453734 9931 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:20.452898 10008 raft_consensus.cc:1240] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:20.460575 10010 raft_consensus.cc:2393] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 7: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:20.464160 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.465493 10011 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:20.467108 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:20.468969 10012 raft_consensus.cc:1240] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:20.472550 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:20.473707 9932 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:20.477100 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:20.478044 9930 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:20.489760 10012 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: []
W20260430 07:59:20.490993 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 164519us
I20260430 07:59:20.489068 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:20.491663 9977 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.662s (SENT) [suppressed 24 similar messages]
W20260430 07:59:20.522348 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.498226 10011 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:20.509423 9859 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:20.501785 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.510228 9860 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:20.493521 10004 raft_consensus.cc:1240] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:20.524986 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:20.494482 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.530725 9930 raft_consensus.cc:3060] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:20.494313 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:20.496729 10007 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:20.494015 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:20.537462 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:20.537864 10006 raft_consensus.cc:1275] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260430 07:59:20.496906 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.499135 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.538882 9978 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:20.539443 10005 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:20.512411 9977 leader_election.cc:336] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.707s (SENT)
I20260430 07:59:20.540488 9977 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:20.542137 9859 raft_consensus.cc:1275] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:20.541873 10534 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:20.549532 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.550518 10007 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 3.
W20260430 07:59:20.492385 12256 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.379s user 0.011s sys 0.016s
W20260430 07:59:20.552634 9828 connection.cc:254] RPC call timeout handler was delayed by -3.675812016000009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15080000000000002s for a response
I20260430 07:59:20.546589 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.556532 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:20.562090 9828 connection.cc:254] RPC call timeout handler was delayed by -3.6614120159999857s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
I20260430 07:59:20.563973 10005 raft_consensus.cc:1275] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:20.564379 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:20.489204 12261 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:20.430473 12180 consensus_queue.cc:1059] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.107s
I20260430 07:59:20.563803 9937 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:20.569228 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:20.570263 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:20.566071 9828 connection.cc:254] RPC call timeout handler was delayed by -3.6398120160000076s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
I20260430 07:59:20.571961 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:20.488878 12246 raft_consensus.cc:2804] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election won for term 9
I20260430 07:59:20.577968 9938 raft_consensus.cc:2393] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:20.577823 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.580637 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:20.578321 9828 connection.cc:254] RPC call timeout handler was delayed by -3.5714120160000107s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16240000000000002s for a response
I20260430 07:59:20.581182 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.581467 10006 raft_consensus.cc:3055] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:20.582026 10006 raft_consensus.cc:740] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:20.580229 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:20.582118 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.582644 9932 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:20.581696 9828 connection.cc:254] RPC call timeout handler was delayed by -3.5597120159999918s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
W20260430 07:59:20.583500 9828 connection.cc:254] RPC call timeout handler was delayed by -3.4625120160000051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17450000000000002s for a response
W20260430 07:59:20.559185 12256 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.446s user 0.011s sys 0.016s
I20260430 07:59:20.582183 10008 raft_consensus.cc:2410] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260430 07:59:20.583844 9828 connection.cc:254] RPC call timeout handler was delayed by -3.4490120159999833s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17600000000000002s for a response
W20260430 07:59:20.585772 9828 connection.cc:254] RPC call timeout handler was delayed by -3.4184120159999907s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1794s for a response
W20260430 07:59:20.586066 9828 connection.cc:254] RPC call timeout handler was delayed by -3.3491120159999923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
W20260430 07:59:20.586359 9828 connection.cc:254] RPC call timeout handler was delayed by -3.3491120159999923s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
W20260430 07:59:20.586623 9828 connection.cc:254] RPC call timeout handler was delayed by -3.2339120159999766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
W20260430 07:59:20.586902 9828 connection.cc:254] RPC call timeout handler was delayed by -3.1862120160000131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20520000000000002s for a response
W20260430 07:59:20.587177 9828 connection.cc:254] RPC call timeout handler was delayed by -3.1862120160000131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20520000000000002s for a response
W20260430 07:59:20.587442 9828 connection.cc:254] RPC call timeout handler was delayed by -3.0134120159999611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22440000000000004s for a response
W20260430 07:59:20.587705 9828 connection.cc:254] RPC call timeout handler was delayed by -2.7785120159999792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2505s for a response
W20260430 07:59:20.587973 9828 connection.cc:254] RPC call timeout handler was delayed by -2.5706120159999841s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2736s for a response
W20260430 07:59:20.588231 9828 connection.cc:254] RPC call timeout handler was delayed by -2.515712016000009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2797s for a response
W20260430 07:59:20.588487 9828 connection.cc:254] RPC call timeout handler was delayed by -2.4905120159999683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28250000000000003s for a response
W20260430 07:59:20.588742 9828 connection.cc:254] RPC call timeout handler was delayed by -2.4896120159999668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2826s for a response
W20260430 07:59:20.589007 9828 connection.cc:254] RPC call timeout handler was delayed by -2.0603120160000117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33030000000000004s for a response
I20260430 07:59:20.553289 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:20.557345 12242 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:20.592131 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.594409 9930 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
W20260430 07:59:20.596530 9828 connection.cc:254] RPC call timeout handler was delayed by -1.6913120159999835s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3713s for a response
I20260430 07:59:20.597905 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:20.598903 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.565088 12341 raft_consensus.cc:2804] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election won for term 6
W20260430 07:59:20.604219 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.624297 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 82452.6us
I20260430 07:59:20.622262 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:20.625674 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:20.627560 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.626945 9937 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:20.635499 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 81630.5us
I20260430 07:59:20.602895 12203 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:59:20.635094 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:20.641417 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 43627.6us
I20260430 07:59:20.643406 9938 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:20.635596 10006 consensus_queue.cc:260] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.583827 12261 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.646582 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:20.647905 12261 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:20.649583 10006 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:20.636672 12180 raft_consensus.cc:3055] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:20.657311 10005 raft_consensus.cc:1275] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260430 07:59:20.636065 12246 raft_consensus.cc:697] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 9 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:20.642485 12341 raft_consensus.cc:697] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:20.635061 12242 raft_consensus.cc:697] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:20.627341 12354 consensus_queue.cc:1059] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260430 07:59:20.687559 10005 raft_consensus.cc:1275] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:20.670468 9828 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:20.663442 12239 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:20.693372 12296 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:20.698828 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:20.730634 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 60246.7us
I20260430 07:59:20.698580 12242 consensus_queue.cc:237] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.699833 12341 consensus_queue.cc:237] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:20.702332 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.633776 12256 consensus_queue.cc:1048] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
W20260430 07:59:20.703879 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.655058 12180 raft_consensus.cc:740] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:20.740908 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:20.743819 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.748385 9934 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:20.754113 12399 log.cc:927] Time spent T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20260430 07:59:20.761528 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.764264 9828 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:20.772507 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.772557 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.776705 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 264291us
I20260430 07:59:20.781173 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:20.781960 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.783342 12388 log.cc:927] Time spent T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20260430 07:59:20.737628 12239 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:20.787207 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:20.791316 9930 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:20.791460 9977 leader_election.cc:336] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.708s (SENT)
I20260430 07:59:20.788307 9828 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:20.790455 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.708603 12390 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:20.669368 12391 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:20.784648 12281 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:20.807206 10006 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
W20260430 07:59:20.806190 9977 leader_election.cc:336] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.753s (SENT)
W20260430 07:59:20.813617 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.818652 9977 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:20.822849 9934 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:20.844585 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 290505us
I20260430 07:59:20.807799 9937 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:20.852713 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:20.852839 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.853443 9934 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:20.853962 10008 raft_consensus.cc:2410] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260430 07:59:20.854753 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.666529 12393 consensus_queue.cc:1048] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.187s
I20260430 07:59:20.877765 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.878774 10006 raft_consensus.cc:2410] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260430 07:59:20.796680 12303 raft_consensus.cc:476] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Not starting pre-election -- already a leader
I20260430 07:59:20.878298 9934 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
W20260430 07:59:20.900962 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 54750.2us
W20260430 07:59:20.885519 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 79332.8us
W20260430 07:59:20.904198 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.885960 9903 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:20.886762 10010 raft_consensus.cc:2468] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:20.887827 9938 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:20.890326 12398 log.cc:927] Time spent T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260430 07:59:20.890826 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1638) took 2349 ms. Trace:
I20260430 07:59:20.896390 9863 raft_consensus.cc:1275] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:20.807178 12296 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:20.908480 9930 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:20.911338 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.922576 9828 leader_election.cc:400] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:20.923343 9828 leader_election.cc:403] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260430 07:59:20.911139 9856 rpcz_store.cc:276] 0430 07:59:18.541323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:18.614305 (+ 72982us) service_pool.cc:225] Handling call
0430 07:59:18.646941 (+ 32636us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:19.680611 (+1033670us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:20.890544 (+1209933us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400208d54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:20.890720 (+ 176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2355612928}
I20260430 07:59:20.930688 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:20.931969 10006 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:20.934892 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:20.935902 10008 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
W20260430 07:59:20.940296 9977 leader_election.cc:336] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.555s (SENT)
W20260430 07:59:20.937386 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 188627us
I20260430 07:59:20.944481 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 295) took 1418 ms. Trace:
I20260430 07:59:20.940927 9977 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:20.779882 12246 consensus_queue.cc:237] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.945960 9935 rpcz_store.cc:276] 0430 07:59:19.526121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:19.526292 (+ 171us) service_pool.cc:225] Handling call
0430 07:59:19.526717 (+ 425us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:19.776715 (+249998us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:19.776757 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:19.806496 (+ 29739us) log.cc:844] Serialized 26 byte log entry
0430 07:59:20.094358 (+287862us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:20.094395 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:20.094472 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:20.944257 (+849785us) raft_consensus.cc:1685] finished
0430 07:59:20.944278 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:20.944421 (+ 143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99866,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":17939,"thread_start_us":340309,"threads_started":2,"wal-append.queue_time_us":582363}
W20260430 07:59:20.916343 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.951076 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.952147 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.952556 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:20.978360 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.816532 12180 consensus_queue.cc:260] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:20.917577 12296 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:20.944460 12386 log.cc:927] Time spent T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.269s user 0.000s sys 0.001s
I20260430 07:59:20.917941 12252 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:21.007694 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 67452.3us
W20260430 07:59:21.009017 9977 leader_election.cc:336] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.637s (SENT)
I20260430 07:59:21.009398 9977 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:20.994906 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.995302 9932 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:20.945147 12355 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:20.909502 12408 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260430 07:59:20.991861 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.022449 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.966588 12049 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:20.996831 12409 log.cc:927] Time spent T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
W20260430 07:59:21.021680 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 69081.7us
W20260430 07:59:21.059737 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.988346 12411 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:21.002054 12296 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:21.012180 12203 consensus_queue.cc:1059] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260430 07:59:21.018721 12252 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:21.053026 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:20.953094 12402 consensus_queue.cc:1048] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
W20260430 07:59:21.022871 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 147280us
I20260430 07:59:21.123353 12408 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:21.006528 12196 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:21.118913 9828 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:21.053026 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.138077 12404 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.118s user 0.000s sys 0.000s
W20260430 07:59:21.138986 12412 log.cc:927] Time spent T 19c3ca044c3d4377b31c4d07c513415c P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260430 07:59:21.142410 12357 log.cc:927] Time spent T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 1.474s user 0.006s sys 0.020s
I20260430 07:59:21.144081 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1657) took 1498 ms. Trace:
W20260430 07:59:21.085767 12403 log.cc:927] Time spent T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260430 07:59:21.077441 10004 consensus_meta.cc:287] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.515s user 0.015s sys 0.009s
W20260430 07:59:21.161900 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.071720 9931 consensus_meta.cc:287] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.615s user 0.007s sys 0.002s
I20260430 07:59:21.200567 9931 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:21.163070 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1646) took 2025 ms. Trace:
I20260430 07:59:21.125062 12180 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:21.203871 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:21.176954 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 58264.5us
I20260430 07:59:21.177775 9903 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:21.180233 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:21.181399 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:44006 (request call id 575) took 1515 ms. Trace:
I20260430 07:59:21.206789 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:21.207549 10010 raft_consensus.cc:3060] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:21.207553 10004 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
W20260430 07:59:21.192592 11626 consensus_meta.cc:287] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.756s user 0.012s sys 0.000s
I20260430 07:59:21.193256 12408 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:21.195062 12252 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:21.178879 12239 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:21.221467 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:21.201630 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.201453 9861 rpcz_store.cc:276] 0430 07:59:19.137932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:19.138059 (+ 127us) service_pool.cc:225] Handling call
0430 07:59:19.138356 (+ 297us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:20.282729 (+1144373us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:21.162754 (+880025us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54001ba654. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:21.163003 (+ 249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2628557568}
I20260430 07:59:21.184314 9862 rpcz_store.cc:276] 0430 07:59:19.645569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:19.669455 (+ 23886us) service_pool.cc:225] Handling call
0430 07:59:19.679460 (+ 10005us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:20.273147 (+593687us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:21.143848 (+870701us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b540029cf54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:21.144014 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1360448768}
I20260430 07:59:21.206089 10009 rpcz_store.cc:276] 0430 07:59:19.653589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:19.669355 (+ 15766us) service_pool.cc:225] Handling call
0430 07:59:21.163894 (+1494539us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b54002e63d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ed470 00007fb56928c31a 00007fb56e25b11e 00007fb5677d7437 00007fb5677d73c2 00007fb5677d72f1 00007fb5677d726c 00007fb5677d6522 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:21.168654 (+ 4760us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":80225536}
I20260430 07:59:21.217510 10010 raft_consensus.cc:2468] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:21.214018 9828 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:21.115988 12049 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:21.223425 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.224145 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 215103us
W20260430 07:59:21.224840 9977 leader_election.cc:336] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.644s (SENT)
I20260430 07:59:21.225200 9977 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:21.247663 9828 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:21.212396 12393 raft_consensus.cc:2764] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260430 07:59:21.273973 9828 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:21.232054 12252 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:21.267558 12347 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:21.219384 11626 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:21.278251 12349 raft_consensus.cc:2764] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260430 07:59:21.280381 12414 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:21.289552 12355 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
W20260430 07:59:21.328738 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.333048 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.228885 12408 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:21.365895 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 141035us
W20260430 07:59:21.366853 9977 leader_election.cc:336] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.842s (SENT)
I20260430 07:59:21.358673 10003 raft_consensus.cc:2368] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8.
I20260430 07:59:21.300949 12347 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:21.374120 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:21.383644 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.411247 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.380126 12347 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:21.416411 12422 consensus_queue.cc:1059] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260430 07:59:21.381997 12360 consensus_queue.cc:1059] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260430 07:59:21.388475 12340 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:21.423229 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.428297 12422 raft_consensus.cc:3055] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:21.428835 12422 raft_consensus.cc:740] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:21.430092 12422 consensus_queue.cc:260] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:21.440327 12422 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:21.440909 9861 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:21.442970 12385 consensus_queue.cc:1059] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260430 07:59:21.444136 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.444584 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.445004 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.450443 9977 leader_election.cc:336] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.881s (SENT)
I20260430 07:59:21.384783 12242 consensus_queue.cc:1059] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.074s
I20260430 07:59:21.462303 12242 raft_consensus.cc:3055] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:21.462711 12242 raft_consensus.cc:740] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:21.463747 12242 consensus_queue.cc:260] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:21.478720 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.465062 12242 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:21.493798 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 272568us
I20260430 07:59:21.495700 9903 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:21.496987 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 123179us
I20260430 07:59:21.499974 9903 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:21.503229 9828 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:21.508198 9903 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:21.469614 12414 raft_consensus.cc:3055] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:21.516846 9862 raft_consensus.cc:1275] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260430 07:59:21.519271 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 107008us
I20260430 07:59:21.421940 12340 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:21.523414 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.527627 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 141777us
W20260430 07:59:21.531391 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 75933us
W20260430 07:59:21.533936 9977 leader_election.cc:336] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.957s (SENT)
I20260430 07:59:21.539302 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:21.541013 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.523250 12414 raft_consensus.cc:740] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:21.532740 12411 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:59:21.558705 9903 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:21.560820 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 57665.6us
I20260430 07:59:21.576721 12414 consensus_queue.cc:260] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:21.576896 12411 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:21.578210 9828 leader_election.cc:400] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9.
I20260430 07:59:21.579061 12414 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 8
I20260430 07:59:21.580404 9828 leader_election.cc:403] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260430 07:59:21.580705 12411 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:21.556619 9860 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260430 07:59:21.544044 12355 raft_consensus.cc:697] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:21.587932 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.599435 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:21.603534 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 137061us
I20260430 07:59:21.603667 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:21.604645 10004 raft_consensus.cc:2410] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260430 07:59:21.600406 12424 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:21.614656 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:21.615581 10006 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:21.619426 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.622854 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:21.623997 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.624123 10009 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
W20260430 07:59:21.637099 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 59531.8us
I20260430 07:59:21.638413 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:21.590708 12246 consensus_queue.cc:1048] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
W20260430 07:59:21.621559 12419 log.cc:927] Time spent T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.319s user 0.000s sys 0.001s
W20260430 07:59:21.647503 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 126943us
I20260430 07:59:21.618507 12049 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:21.657254 9977 leader_election.cc:336] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.973s (SENT)
I20260430 07:59:21.658810 9977 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:59:21.668524 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.673924 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.673985 10006 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:21.617766 12424 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:21.687639 12395 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.986s user 0.005s sys 0.000s
I20260430 07:59:21.694504 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:21.688906 9859 raft_consensus.cc:1275] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260430 07:59:21.689459 12418 log.cc:927] Time spent T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
W20260430 07:59:21.690846 12429 log.cc:927] Time spent T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
W20260430 07:59:21.691462 12426 log.cc:927] Time spent T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.136s user 0.001s sys 0.001s
I20260430 07:59:21.649940 11626 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:21.692945 12407 log.cc:927] Time spent T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.361s user 0.000s sys 0.001s
I20260430 07:59:21.708498 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:21.726639 9929 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:21.709842 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:21.728327 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 71076.5us
I20260430 07:59:21.709672 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:21.728966 9977 leader_election.cc:336] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.077s (SENT)
I20260430 07:59:21.729221 9935 raft_consensus.cc:2376] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6. Re-sending same reply.
I20260430 07:59:21.712157 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:21.731091 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:21.731897 9937 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:21.712390 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:21.713788 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.733235 9938 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
W20260430 07:59:21.733796 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.715560 9932 raft_consensus.cc:1275] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:21.717183 12431 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:21.736886 12369 raft_consensus.cc:3055] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:21.737432 12369 raft_consensus.cc:740] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:21.738924 12453 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260430 07:59:21.718133 12430 log.cc:927] Time spent T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
I20260430 07:59:21.739696 12453 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:21.740216 12453 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:21.717808 12435 log.cc:927] Time spent T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260430 07:59:21.742542 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:21.743270 9930 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 7
W20260430 07:59:21.743767 12180 consensus_meta.cc:287] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.530s user 0.006s sys 0.006s
I20260430 07:59:21.714679 12203 consensus_queue.cc:1048] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260430 07:59:21.746927 9936 raft_consensus.cc:1275] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260430 07:59:21.747056 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:21.747673 9935 raft_consensus.cc:3060] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:21.682117 12355 consensus_queue.cc:237] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:21.748068 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:21.750635 12449 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:21.751125 12449 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:21.751497 12449 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:21.738408 12369 consensus_queue.cc:260] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:21.752996 12369 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:21.742563 12423 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:21.764633 12423 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:21.766376 12420 log.cc:927] Time spent T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.354s user 0.001s sys 0.000s
I20260430 07:59:21.766996 12423 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:21.753158 12347 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:21.769836 12347 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:21.770469 9903 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:21.756186 12408 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:21.759718 12455 consensus_queue.cc:1059] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:21.719375 9934 raft_consensus.cc:1275] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260430 07:59:21.649942 12411 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:21.770982 9861 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: []
W20260430 07:59:21.793766 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 64823.1us
I20260430 07:59:21.794588 9934 pending_rounds.cc:85] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 2
I20260430 07:59:21.795069 9934 pending_rounds.cc:107] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260430 07:59:21.795339 9934 raft_consensus.cc:2889] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:21.772733 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1051) took 1229 ms. Trace:
I20260430 07:59:21.771737 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:21.799643 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 178133us
I20260430 07:59:21.796548 9933 rpcz_store.cc:276] 0430 07:59:20.542887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:20.544237 (+ 1350us) service_pool.cc:225] Handling call
0430 07:59:20.567653 (+ 23416us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:20.995295 (+427642us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:20.995327 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:21.017687 (+ 22360us) log.cc:844] Serialized 52 byte log entry
0430 07:59:21.260965 (+243278us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:21.261000 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:21.261173 (+ 173us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:21.772481 (+511308us) raft_consensus.cc:1685] finished
0430 07:59:21.772510 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:21.772673 (+ 163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11578,"raft.queue_time_us":27099,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":25265,"thread_start_us":578519,"threads_started":2,"wal-append.queue_time_us":329628}
W20260430 07:59:21.809417 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.772131 12296 consensus_meta.cc:287] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.615s user 0.011s sys 0.000s
I20260430 07:59:21.813084 9935 raft_consensus.cc:2468] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:21.817739 9930 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
W20260430 07:59:21.812310 9977 leader_election.cc:336] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.233s (SENT)
I20260430 07:59:21.820094 9977 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:21.776999 12424 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:21.825649 12296 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:21.827086 12450 raft_consensus.cc:3055] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:21.829610 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:21.740233 12340 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:21.850893 12296 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:21.852511 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.830463 12298 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260430 07:59:21.865494 9855 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:21.834456 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:21.843065 10003 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1699) took 1402 ms (client timeout 1785 ms). Trace:
W20260430 07:59:21.870152 10003 rpcz_store.cc:269] 0430 07:59:20.440376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:20.440498 (+ 122us) service_pool.cc:225] Handling call
0430 07:59:21.842795 (+1402297us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b540038cf54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:21.842990 (+ 195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2108341120}
I20260430 07:59:21.773051 12390 consensus_queue.cc:1048] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
W20260430 07:59:21.872372 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 60079.6us
W20260430 07:59:21.873082 9977 leader_election.cc:336] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.470s (SENT)
I20260430 07:59:21.873493 9977 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:21.877418 12441 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:21.878171 12441 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:21.905416 11626 consensus_queue.cc:1059] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:21.770596 12347 consensus_queue.cc:1059] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260430 07:59:21.907960 12347 raft_consensus.cc:3055] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:21.908356 12347 raft_consensus.cc:740] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:21.909190 12347 consensus_queue.cc:260] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:21.910705 12347 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:21.853593 9936 raft_consensus.cc:3055] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:21.912034 9936 raft_consensus.cc:740] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:21.852298 12378 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:21.854542 9863 raft_consensus.cc:1240] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:21.857504 12256 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:59:21.822086 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:21.798632 12336 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9.
W20260430 07:59:21.885607 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.915213 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.806694 12423 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:21.892110 12049 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260430 07:59:21.892235 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 57847.8us
W20260430 07:59:21.894681 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 124310us
I20260430 07:59:21.845161 12450 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:21.913599 9929 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
W20260430 07:59:21.913789 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.915628 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.911276 12180 raft_consensus.cc:2749] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260430 07:59:21.924952 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.928830 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.928504 9977 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:21.935941 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.926378 9936 consensus_queue.cc:260] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:21.942406 9903 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:21.950683 9936 raft_consensus.cc:3060] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:21.980280 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:21.936349 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 384749us
I20260430 07:59:21.982633 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:21.983793 10004 raft_consensus.cc:2410] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260430 07:59:21.971086 9858 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1649) took 2167 ms (client timeout 1910 ms). Trace:
W20260430 07:59:21.985720 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 323239us
W20260430 07:59:21.973742 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 52073us
I20260430 07:59:21.986261 9936 raft_consensus.cc:1275] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260430 07:59:21.968780 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:21.985946 9858 rpcz_store.cc:269] 0430 07:59:19.777399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:19.879324 (+101925us) service_pool.cc:225] Handling call
0430 07:59:21.944902 (+2065578us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b540010c054. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:21.945061 (+ 159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1560852864}
I20260430 07:59:22.014528 10008 raft_consensus.cc:1275] T e51196f9c4fb44bf8f6296f5394417bb P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260430 07:59:22.030603 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.977573 12421 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:22.032295 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 48097.2us
W20260430 07:59:22.032912 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.965612 11769 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:21.968489 12408 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:22.038389 9856 raft_consensus.cc:1240] T 38bcd78c11574f1a9e016f9bb9365ca5 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:22.041762 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:21.970980 12441 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:22.045194 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:22.047847 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 59105.5us
I20260430 07:59:22.053335 10533 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:22.059491 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:22.061205 9828 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:22.068145 9903 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:21.969583 12453 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:22.078554 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:22.061434 12413 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:22.121333 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 73552.5us
I20260430 07:59:22.073210 11769 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:22.130091 9903 leader_election.cc:400] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:22.130720 9903 leader_election.cc:403] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260430 07:59:22.131374 12432 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:22.135367 9903 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:22.136225 12432 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:22.136646 12432 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:22.044409 12421 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:22.067482 12379 consensus_queue.cc:1048] T e51196f9c4fb44bf8f6296f5394417bb P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20260430 07:59:22.058509 12393 consensus_queue.cc:1059] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260430 07:59:22.143632 12453 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:22.146984 12449 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:22.148964 12196 raft_consensus.cc:3055] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
W20260430 07:59:22.151168 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:22.155894 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:22.170476 12421 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:22.178275 12449 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:22.149677 12196 raft_consensus.cc:740] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:22.181636 12196 consensus_queue.cc:260] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:22.136997 12432 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:22.191239 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:22.192257 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1712) took 1453 ms. Trace:
I20260430 07:59:22.192858 10005 rpcz_store.cc:276] 0430 07:59:20.739931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:20.740222 (+ 291us) service_pool.cc:225] Handling call
0430 07:59:20.740697 (+ 475us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:20.758334 (+ 17637us) raft_consensus.cc:1534] Early marking committed up to index 3
0430 07:59:20.876459 (+118125us) log.cc:844] Serialized 14 byte log entry
0430 07:59:21.765652 (+889193us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:21.857454 (+ 91802us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b1800030160. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb568e2a85c 00007fb5661c8c0b 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e2a95 00007fb56927b440 00007fb56928daf6 00007fb56929e667
0430 07:59:21.857658 (+ 204us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:21.858218 (+ 560us) log.cc:844] Serialized 26 byte log entry
0430 07:59:21.858397 (+ 179us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:21.862786 (+ 4389us) log.cc:844] Serialized 14 byte log entry
0430 07:59:22.175055 (+312269us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b1800030160. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb566159e1a 00007fb56f7f25c9 00007fb568e39110 00007fb568e2c633 00007fb568e2208a 00007fb568e225da 00007fb56928de0a 00007fb56929d49f 00007fb56929d418
0430 07:59:22.191758 (+ 16703us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:22.191882 (+ 124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:22.191943 (+ 61us) raft_consensus.cc:1685] finished
0430 07:59:22.191964 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:22.192171 (+ 207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":325330,"prepare.queue_time_us":315306,"prepare.run_cpu_time_us":5715,"prepare.run_wall_time_us":30154,"raft.queue_time_us":35753,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":18143,"spinlock_wait_cycles":60117888,"thread_start_us":398255,"threads_started":3,"wal-append.queue_time_us":353471}
I20260430 07:59:22.196134 9930 raft_consensus.cc:2410] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20260430 07:59:22.203814 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:21.965186 12450 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:22.206346 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 46291.7us
I20260430 07:59:22.054462 12427 consensus_queue.cc:1048] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260430 07:59:22.214599 10011 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
W20260430 07:59:22.220371 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 163562us
W20260430 07:59:22.222350 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 87100.8us
I20260430 07:59:22.223711 9903 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.213397 12450 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:22.209934 12196 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:22.230610 12369 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:22.255333 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:22.255677 10012 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 1081) took 1761 ms (client timeout 2129 ms). Trace:
W20260430 07:59:22.262959 10012 rpcz_store.cc:269] 0430 07:59:20.494085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:20.494237 (+ 152us) service_pool.cc:225] Handling call
0430 07:59:22.255469 (+1761232us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x7b5400147b54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:22.255620 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2268026112}
W20260430 07:59:22.263787 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:22.258914 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1660) took 1264 ms. Trace:
I20260430 07:59:22.264959 9863 rpcz_store.cc:276] 0430 07:59:20.994637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:20.994871 (+ 234us) service_pool.cc:225] Handling call
0430 07:59:20.995173 (+ 302us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:21.913126 (+917953us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:22.258701 (+345575us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b540012edd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:22.258856 (+ 155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1970837760}
W20260430 07:59:22.268638 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 45052.2us
W20260430 07:59:22.263199 12357 log.cc:927] Time spent T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260430 07:59:22.274507 9903 leader_election.cc:400] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:22.274885 9903 leader_election.cc:403] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20260430 07:59:22.280526 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 288172us
W20260430 07:59:22.287151 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:22.287922 9903 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:59:22.288122 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:22.278264 12369 raft_consensus.cc:2749] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:22.289582 12410 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:22.293588 12443 log.cc:927] Time spent T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.647s user 0.004s sys 0.001s
I20260430 07:59:22.297760 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:22.300251 9854 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1663) took 2019 ms (client timeout 1793 ms). Trace:
W20260430 07:59:22.300704 9854 rpcz_store.cc:269] 0430 07:59:20.280348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:20.378151 (+ 97803us) service_pool.cc:225] Handling call
0430 07:59:22.299943 (+1921792us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b54003bfbd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:22.300175 (+ 232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":921652608}
W20260430 07:59:22.309347 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:22.311743 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:22.313589 10007 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
W20260430 07:59:22.314213 10007 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1708) took 1712 ms (client timeout 1521 ms). Trace:
W20260430 07:59:22.316294 10007 rpcz_store.cc:269] 0430 07:59:20.602121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:20.627297 (+ 25176us) service_pool.cc:225] Handling call
0430 07:59:22.297637 (+1670340us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b58000a2098. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8a3e 000000000035a3c9 00007fb56e1e08e0 00007fb56e2a9784 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:22.314123 (+ 16486us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4585344}
W20260430 07:59:22.327133 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:22.336586 9903 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:22.340955 12284 raft_consensus.cc:2764] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260430 07:59:22.341881 12462 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
W20260430 07:59:22.369689 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:22.371521 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:22.377174 10011 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 1086) took 1831 ms (client timeout 1587 ms). Trace:
W20260430 07:59:22.377839 10011 rpcz_store.cc:269] 0430 07:59:20.546009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:20.546334 (+ 325us) service_pool.cc:225] Handling call
0430 07:59:22.376874 (+1830540us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b540018e2d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:22.377063 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3640958336}
I20260430 07:59:22.387876 9903 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:22.409952 9903 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.411700 9903 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:22.413945 9903 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.424219 12369 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:22.437151 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:44006 (request call id 576) took 1166 ms. Trace:
I20260430 07:59:22.437886 10010 rpcz_store.cc:276] 0430 07:59:21.270453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:21.270613 (+ 160us) service_pool.cc:225] Handling call
0430 07:59:22.436310 (+1165697us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x7b54000cd354. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ed470 00007fb56928c31a 00007fb56e25b11e 00007fb5677d7437 00007fb5677d73c2 00007fb5677d72f1 00007fb5677d726c 00007fb5677d6522 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:22.437040 (+ 730us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":95724928}
I20260430 07:59:22.406672 12424 consensus_queue.cc:1059] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260430 07:59:22.389735 12414 consensus_queue.cc:1048] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260430 07:59:22.421893 12410 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:22.423282 12441 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:22.415001 12458 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:22.422147 12275 consensus_queue.cc:1048] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.032s
I20260430 07:59:22.422202 11769 consensus_queue.cc:1048] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.032s
I20260430 07:59:22.479413 11769 consensus_queue.cc:1243] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer 48e81934d52a4e93be5171ceea193032 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:59:22.421942 12423 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
W20260430 07:59:22.486380 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 425117us
I20260430 07:59:22.487914 9828 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:22.489248 9828 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:22.493556 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 46457.4us
I20260430 07:59:22.453176 12410 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:59:22.511291 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:22.508737 12465 raft_consensus.cc:2804] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
W20260430 07:59:22.512477 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 98552.1us
I20260430 07:59:22.532826 9903 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.534152 9903 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.535671 9903 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.507371 12355 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
W20260430 07:59:22.544821 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 55665.9us
I20260430 07:59:22.544968 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:22.546293 9862 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
W20260430 07:59:22.549372 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 137914us
I20260430 07:59:22.550196 9828 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:22.550189 9903 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.533222 12410 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:22.557050 9903 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.561997 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:22.544337 12441 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:22.538738 12284 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:22.568090 9857 raft_consensus.cc:1240] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:22.546296 12424 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:19.382728 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:59:22.563449 9903 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.573113 9977 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:22.552227 12347 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:22.511268 12408 consensus_queue.cc:1048] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.043s
I20260430 07:59:22.536599 12378 consensus_queue.cc:1048] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260430 07:59:22.558460 11769 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:22.555581 12423 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:22.596585 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 46463us
I20260430 07:59:22.561426 12465 raft_consensus.cc:697] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:22.607951 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 44572.4us
I20260430 07:59:22.613500 9903 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:22.609896 9828 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:22.609007 12441 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:22.650103 9903 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:22.580161 12355 raft_consensus.cc:697] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:22.659380 9903 leader_election.cc:400] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8.
I20260430 07:59:22.671092 9903 leader_election.cc:403] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260430 07:59:22.669838 9828 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:22.672092 12458 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:22.681797 12458 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:22.682226 12458 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:22.682098 12465 consensus_queue.cc:237] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:22.686326 12355 consensus_queue.cc:237] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:22.610049 12449 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:22.599493 12347 raft_consensus.cc:697] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:22.706954 9854 raft_consensus.cc:1240] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
W20260430 07:59:22.708009 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 134915us
I20260430 07:59:22.608610 12476 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:22.645844 12390 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:22.720162 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 61094us
W20260430 07:59:22.722291 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 52535.9us
I20260430 07:59:22.728641 9828 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:22.721455 9977 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:22.725858 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1667) took 1185 ms. Trace:
I20260430 07:59:22.732565 9861 rpcz_store.cc:276] 0430 07:59:21.540469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:21.540672 (+ 203us) service_pool.cc:225] Handling call
0430 07:59:21.541002 (+ 330us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:21.794564 (+253562us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:22.725642 (+931078us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b540029e854. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:22.725800 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":528328960}
I20260430 07:59:22.690907 12424 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:22.611720 12275 consensus_queue.cc:1059] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
I20260430 07:59:22.751995 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:22.718021 12440 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:22.629024 12423 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:22.771556 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 220828us
W20260430 07:59:22.808573 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 81305.3us
I20260430 07:59:22.809515 10011 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260430 07:59:22.741437 12441 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:22.753957 12196 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:22.815197 10012 raft_consensus.cc:1240] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
W20260430 07:59:22.815650 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 215879us
I20260430 07:59:22.772290 12408 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:22.813076 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:22.825568 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 52875.1us
I20260430 07:59:22.823403 9828 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.808987 12390 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:22.771519 12449 raft_consensus.cc:697] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:22.787748 12393 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:22.830513 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 80083.7us
I20260430 07:59:22.839869 9828 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:22.786610 12450 consensus_queue.cc:1059] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
W20260430 07:59:22.840764 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 244041us
I20260430 07:59:22.845712 9903 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:22.848124 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:22.917156 10010 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:22.917414 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 77723.4us
I20260430 07:59:22.921151 9828 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:22.925429 10008 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260430 07:59:22.927060 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:22.928187 10004 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:22.821349 12479 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:22.932508 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:22.935364 10003 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:22.858736 12427 consensus_queue.cc:1059] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260430 07:59:22.950807 10004 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:22.853576 12256 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:22.954677 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:22.920837 12439 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:22.828002 12196 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:22.957075 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:22.959978 10006 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:22.960397 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:22.961995 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 135676us
I20260430 07:59:22.962769 9977 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:22.822039 12458 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:22.830086 12440 consensus_queue.cc:1048] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.124s
I20260430 07:59:22.961016 10004 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:22.934639 12476 raft_consensus.cc:3055] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:22.993521 10003 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:22.956427 12281 raft_consensus.cc:2764] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260430 07:59:22.997751 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:22.996021 10010 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:22.963660 12390 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:23.001830 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:22.858314 12347 consensus_queue.cc:237] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.018561 10003 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260430 07:59:22.989045 12424 raft_consensus.cc:697] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:59:23.003715 12432 consensus_meta.cc:287] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.817s user 0.016s sys 0.000s
I20260430 07:59:22.965101 12256 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:22.965205 12393 consensus_queue.cc:1059] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260430 07:59:23.038062 12432 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:22.996733 12476 raft_consensus.cc:740] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:23.043114 12256 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:23.047111 10006 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:23.047418 10003 raft_consensus.cc:1240] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260430 07:59:23.052855 10004 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:23.005416 12487 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:23.093041 10004 raft_consensus.cc:1240] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260430 07:59:23.100376 10003 pending_rounds.cc:85] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 2
I20260430 07:59:23.100725 10003 pending_rounds.cc:107] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260430 07:59:23.100955 10003 raft_consensus.cc:2889] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260430 07:59:23.101943 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:23.114609 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 254745us
W20260430 07:59:23.115557 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 340070us
W20260430 07:59:23.115927 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 156848us
W20260430 07:59:23.111263 12486 log.cc:927] Time spent T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.188s user 0.003s sys 0.000s
W20260430 07:59:23.107797 12492 log.cc:927] Time spent T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.067s user 0.003s sys 0.001s
I20260430 07:59:23.109849 12454 raft_consensus.cc:2764] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Leader pre-election decision vote started in defunct term 6: won
I20260430 07:59:23.212713 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:23.085893 12281 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:23.139900 12423 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:23.003067 12449 consensus_queue.cc:237] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:23.205013 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 359349us
I20260430 07:59:23.205610 10006 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260430 07:59:23.206466 9978 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:23.205977 12453 raft_consensus.cc:3055] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:23.210166 9930 raft_consensus.cc:1275] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:23.221467 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:23.208326 9828 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:23.225836 12458 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:23.226411 9862 raft_consensus.cc:1240] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:23.177793 12487 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:23.230996 9929 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
W20260430 07:59:23.231280 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:23.238466 9903 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:23.240399 9903 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:23.242076 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:23.245751 9828 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:23.251255 12485 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:23.254657 12196 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:23.247323 12423 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:23.255378 12423 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:23.255767 12423 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:23.241933 12458 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:23.258008 12458 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.260795 12458 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:23.212070 12275 consensus_queue.cc:1059] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260430 07:59:23.263303 12275 raft_consensus.cc:3055] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:23.263836 12275 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:23.266045 12440 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:23.266795 12440 raft_consensus.cc:697] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:23.267966 12440 consensus_queue.cc:237] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.270265 12369 raft_consensus.cc:3055] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:23.239897 12487 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:23.274788 12487 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:23.270730 12369 raft_consensus.cc:740] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:23.279366 12369 consensus_queue.cc:260] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:23.280718 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 317944us
I20260430 07:59:23.281591 9977 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:23.282299 12402 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:23.282775 9977 leader_election.cc:400] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6.
I20260430 07:59:23.283493 12256 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:23.283869 9977 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:59:23.284276 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 153127us
I20260430 07:59:23.285022 9977 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:59:23.287416 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 58872.7us
W20260430 07:59:23.291352 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 44641.2us
I20260430 07:59:23.230444 12453 raft_consensus.cc:740] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:23.308909 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 63294.8us
I20260430 07:59:23.311096 9828 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:23.288609 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:23.312656 9828 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:23.153282 12476 consensus_queue.cc:260] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.328167 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:23.332855 10568 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:23.330875 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:23.334684 9903 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:23.207091 12424 consensus_queue.cc:237] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.280763 12378 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260430 07:59:23.341723 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 56749.9us
I20260430 07:59:23.344138 9977 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:23.227119 12281 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.307682 11626 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:23.319355 12465 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:23.323380 12303 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
W20260430 07:59:23.463495 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 138005us
I20260430 07:59:23.326277 12450 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:23.307963 12196 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:23.372651 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:23.460294 10006 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:23.308128 12458 consensus_queue.cc:1059] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260430 07:59:23.474036 10011 raft_consensus.cc:1275] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260430 07:59:23.476569 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 130844us
I20260430 07:59:23.459601 9936 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260430 07:59:23.264868 12275 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:23.465456 12410 consensus_meta.cc:287] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.892s user 0.013s sys 0.001s
I20260430 07:59:23.488333 10006 raft_consensus.cc:1275] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:23.356781 12490 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:23.282785 12402 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.356654 12378 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:59:23.511512 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:23.356204 12502 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:23.482559 12196 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:23.519418 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:23.524859 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 180767us
I20260430 07:59:23.342886 12485 consensus_queue.cc:1059] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.156s
I20260430 07:59:23.478430 12432 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:23.489383 12483 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:23.550750 9977 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:23.501773 12493 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:23.509894 10560 raft_consensus.cc:1240] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260430 07:59:23.489238 12393 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260430 07:59:23.537734 12502 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:23.531260 12378 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:23.607577 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:23.618511 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:23.620445 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:23.620312 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:23.616796 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:23.629504 10011 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
W20260430 07:59:23.631757 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 128260us
I20260430 07:59:23.563586 12421 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:23.632874 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 54453.6us
I20260430 07:59:23.568477 12275 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:23.550316 12196 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:23.643172 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:23.648216 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:23.648245 9828 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:23.649649 9828 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:23.484722 12453 consensus_queue.cc:260] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.651835 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:23.521425 12410 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:23.674381 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 123668us
I20260430 07:59:23.685518 9977 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:23.681934 9933 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:23.626377 12489 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:23.652330 12494 log.cc:927] Time spent T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.429s user 0.001s sys 0.000s
W20260430 07:59:23.689117 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 55458.1us
I20260430 07:59:23.696310 9977 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:23.644650 12479 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:23.638244 12502 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:23.674453 12478 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:23.606604 12440 consensus_queue.cc:1059] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.091s
I20260430 07:59:23.735101 12393 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:23.735318 12440 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:23.735816 12440 raft_consensus.cc:697] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:23.735719 12393 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.601065 12493 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:23.737844 12519 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:23.739326 12493 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:23.577567 12483 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:23.732816 12441 consensus_meta.cc:287] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.914s user 0.012s sys 0.000s
I20260430 07:59:23.741340 12441 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8.
I20260430 07:59:23.742045 12483 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:23.742646 12483 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:23.653200 12423 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.745102 12423 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:23.745483 12451 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:23.746030 12451 raft_consensus.cc:697] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:23.673242 12432 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:23.732751 9933 raft_consensus.cc:1240] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260430 07:59:23.733675 12478 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.756161 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:23.734654 12402 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:23.747074 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 50856.4us
I20260430 07:59:23.747587 9937 raft_consensus.cc:1240] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:23.747702 9904 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:23.759079 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:23.748009 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:23.748149 9938 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:23.537997 12490 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.751163 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:23.761672 9857 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:23.752169 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:23.758910 9977 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:23.706238 12465 raft_consensus.cc:697] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:23.763553 10007 raft_consensus.cc:1275] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:23.761736 9860 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:23.765555 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:23.767721 9904 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:23.745132 12507 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20260430 07:59:23.761215 10533 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:23.769510 9904 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:23.736791 12440 consensus_queue.cc:237] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.778334 9932 raft_consensus.cc:1275] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:23.763406 12493 raft_consensus.cc:2749] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:23.780611 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:23.793615 10012 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:23.771049 12458 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260430 07:59:23.815673 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 56860.5us
I20260430 07:59:23.759801 12432 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.785106 12281 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:23.773506 12450 raft_consensus.cc:697] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:23.785409 12441 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:23.768210 12284 consensus_queue.cc:1059] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
W20260430 07:59:23.832850 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 182183us
I20260430 07:59:23.778331 12523 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:23.789701 12482 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:23.821627 12498 log.cc:927] Time spent T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.304s user 0.001s sys 0.000s
I20260430 07:59:23.707535 12516 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:23.777915 12390 consensus_queue.cc:1059] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260430 07:59:23.835673 9828 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:23.848003 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:23.851245 12303 raft_consensus.cc:697] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:23.852222 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 77095.3us
I20260430 07:59:23.802569 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:23.808250 12518 raft_consensus.cc:2749] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:23.916500 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:23.842907 12506 log.cc:927] Time spent T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.292s user 0.000s sys 0.001s
I20260430 07:59:23.824973 12336 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:23.934832 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:23.935206 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 114474us
I20260430 07:59:23.947455 9977 leader_election.cc:400] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7.
I20260430 07:59:23.947937 9977 leader_election.cc:403] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260430 07:59:23.840648 12479 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:23.955233 10012 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:23.966048 9903 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:23.858254 12393 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:23.771713 12451 consensus_queue.cc:237] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:23.974900 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 184049us
I20260430 07:59:23.976176 9977 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:23.976267 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:23.933990 10560 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
W20260430 07:59:23.981886 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 146284us
I20260430 07:59:23.953488 12441 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:23.967689 12517 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:24.006657 12390 consensus_queue.cc:1059] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:23.970203 12336 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:23.977638 12504 log.cc:927] Time spent T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.340s user 0.000s sys 0.001s
I20260430 07:59:24.008405 12390 consensus_queue.cc:1059] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:59:24.009071 12336 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:24.007067 12517 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.009824 12526 raft_consensus.cc:3055] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:24.010294 12393 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260430 07:59:23.975951 12511 log.cc:927] Time spent T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.295s user 0.001s sys 0.001s
I20260430 07:59:23.968171 12281 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:24.012514 12281 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.013564 10003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1732) took 1055 ms. Trace:
I20260430 07:59:24.014568 12281 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:23.943379 12465 consensus_queue.cc:237] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.991531 9977 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:23.989705 12529 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:23.989300 11626 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:24.001672 12303 consensus_queue.cc:237] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.004727 12432 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:23.968528 12479 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:23.972635 12423 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election won for term 6
W20260430 07:59:23.981709 12520 log.cc:927] Time spent T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.137s user 0.001s sys 0.001s
I20260430 07:59:24.010344 12526 raft_consensus.cc:740] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:24.060341 9977 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:59:24.019558 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 250060us
I20260430 07:59:24.021922 9903 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:23.988476 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.013940 10003 rpcz_store.cc:276] 0430 07:59:22.958050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.099802 (+141752us) service_pool.cc:225] Handling call
0430 07:59:23.100152 (+ 350us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:23.282990 (+182838us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:23.283020 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:23.283604 (+ 584us) log.cc:844] Serialized 52 byte log entry
0430 07:59:23.463303 (+179699us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:23.481042 (+ 17739us) log.cc:844] Serialized 14 byte log entry
0430 07:59:23.503306 (+ 22264us) log.cc:844] Serialized 14 byte log entry
0430 07:59:23.531649 (+ 28343us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:23.531791 (+ 142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:24.013327 (+481536us) raft_consensus.cc:1685] finished
0430 07:59:24.013349 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:24.013502 (+ 153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":362720,"prepare.queue_time_us":17689,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":7714,"raft.queue_time_us":407682,"raft.run_cpu_time_us":517,"raft.run_wall_time_us":29727,"spinlock_wait_cycles":96259200,"thread_start_us":179331,"threads_started":1,"wal-append.queue_time_us":266479}
I20260430 07:59:23.978695 12410 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:23.977995 12458 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:24.038638 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:24.016975 12196 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:24.025702 10008 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s user 0.009s sys 0.002s
I20260430 07:59:23.977355 12523 raft_consensus.cc:697] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:24.063546 9904 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:23.930042 12516 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:23.549825 12495 consensus_queue.cc:1175] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "48e81934d52a4e93be5171ceea193032" responder_term: 7 status { last_received { term: 4 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: [6.3-6.3]" } } last_received_current_leader { term: 0 index: 0 } }
W20260430 07:59:24.071813 12256 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.000s sys 0.010s
W20260430 07:59:24.077968 10008 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.792s user 0.009s sys 0.002s
I20260430 07:59:23.977325 12489 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7.
I20260430 07:59:24.079355 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:24.068045 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.009760 12336 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260430 07:59:24.087102 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:24.089531 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.093583 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.063796 12284 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:24.077117 12432 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260430 07:59:24.111095 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:24.176312 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:24.176999 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 168182us
I20260430 07:59:24.167483 10006 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:24.017797 12450 consensus_queue.cc:237] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:24.105365 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:24.076887 12529 raft_consensus.cc:697] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:24.175122 9929 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
W20260430 07:59:24.085927 12422 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.767s user 0.010s sys 0.018s
W20260430 07:59:24.179658 9854 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.011s sys 0.003s
I20260430 07:59:24.184720 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:24.185644 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:24.186766 10009 raft_consensus.cc:1240] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:24.077432 12423 raft_consensus.cc:697] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:59:24.190451 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 126286us
I20260430 07:59:24.024545 12441 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.059513 12402 consensus_queue.cc:1048] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.104s
W20260430 07:59:24.165820 12256 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s user 0.000s sys 0.011s
I20260430 07:59:24.082316 11626 consensus_queue.cc:1048] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.115s
I20260430 07:59:24.163394 12410 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:24.104965 12493 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:24.080564 12458 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:24.262442 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 199207us
I20260430 07:59:24.286993 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:24.287693 9937 raft_consensus.cc:2376] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8. Re-sending same reply.
W20260430 07:59:24.228339 12422 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.918s user 0.011s sys 0.034s
I20260430 07:59:24.282354 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.190708 12275 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:24.282990 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.175422 12518 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:24.299101 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.281378 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:24.250993 9854 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s user 0.011s sys 0.008s
I20260430 07:59:24.288731 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:24.291620 9934 raft_consensus.cc:2376] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6. Re-sending same reply.
I20260430 07:59:24.175161 12432 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:24.017618 12517 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.017647 12490 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:24.294791 12465 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:24.225996 12478 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:24.301942 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.320286 9934 raft_consensus.cc:2393] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:24.302481 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:24.301158 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1704) took 1013 ms. Trace:
I20260430 07:59:24.255167 12256 raft_consensus.cc:697] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:24.309876 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.312956 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.328090 9933 raft_consensus.cc:2376] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7. Re-sending same reply.
I20260430 07:59:24.305668 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:24.193182 12284 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:24.228660 12489 consensus_queue.cc:1048] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
I20260430 07:59:24.358242 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.322518 12432 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:24.360147 9930 raft_consensus.cc:1275] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:24.321492 12518 raft_consensus.cc:697] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 8 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:24.372696 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.377080 9932 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:24.300575 12493 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:24.305217 12410 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.384373 9933 raft_consensus.cc:1275] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260430 07:59:24.347138 12531 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260430 07:59:24.379006 12378 consensus_meta.cc:287] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.730s user 0.007s sys 0.005s
I20260430 07:59:24.227747 12526 consensus_queue.cc:260] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.379271 12517 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:24.375217 12402 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:24.372682 12532 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:24.403048 9937 raft_consensus.cc:2468] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:24.400610 12378 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.346274 12256 consensus_queue.cc:237] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.312520 12529 consensus_queue.cc:237] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.409477 12524 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:24.409017 12518 consensus_queue.cc:237] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.415192 12517 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.396960 12531 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:24.424582 12531 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:24.425320 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:24.426079 10560 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:24.427313 12424 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:24.427863 12424 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.414590 12493 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:24.358106 12514 consensus_queue.cc:1048] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.355s
I20260430 07:59:24.429577 12524 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.430871 12424 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.431638 12524 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.421684 12532 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.434803 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1063) took 1199 ms. Trace:
I20260430 07:59:24.435582 12532 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.436455 9903 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:24.437657 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.438309 9930 raft_consensus.cc:2376] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7. Re-sending same reply.
I20260430 07:59:24.439206 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.439803 9930 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:24.440570 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.442344 12275 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:24.443215 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.329990 12423 consensus_queue.cc:237] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:24.451303 12508 log.cc:927] Time spent T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.719s user 0.007s sys 0.003s
I20260430 07:59:24.402257 12196 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:24.452854 12458 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.414489 12516 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:24.413839 12441 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.422503 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:24.415069 12402 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.423478 12378 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:24.425482 12517 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.455081 12402 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:24.441089 9930 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:24.456854 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.458415 9930 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:24.445354 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.418815 12479 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:24.443953 9934 raft_consensus.cc:2376] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7. Re-sending same reply.
I20260430 07:59:24.449379 12493 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:24.450008 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:24.448985 12275 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:24.435206 9929 rpcz_store.cc:276] 0430 07:59:23.235338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.235476 (+ 138us) service_pool.cc:225] Handling call
0430 07:59:23.235788 (+ 312us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:24.196922 (+961134us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:24.434537 (+237615us) spinlock_profiling.cc:243] Waited 938 ms on lock 0x7b54001b26d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:24.434743 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2158029056}
I20260430 07:59:24.459539 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.460186 9932 raft_consensus.cc:2376] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8. Re-sending same reply.
I20260430 07:59:24.470876 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.471678 9929 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:24.312250 12523 consensus_queue.cc:237] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.468534 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.485077 9934 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:24.486253 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.487917 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.472757 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.456521 9904 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:24.488909 9929 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:24.490293 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.491276 9929 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:24.492468 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.476075 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:24.481050 12519 consensus_meta.cc:287] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.743s user 0.000s sys 0.012s
I20260430 07:59:24.480748 12493 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.455935 12284 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:24.501751 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 45295.1us
I20260430 07:59:24.481604 12275 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:24.481638 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:24.506237 12531 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.506778 12410 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.507438 10006 raft_consensus.cc:1240] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260430 07:59:24.505460 9904 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:24.509167 12531 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:24.416064 12502 consensus_meta.cc:287] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.660s user 0.008s sys 0.002s
I20260430 07:59:24.512668 12502 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:24.518654 9930 raft_consensus.cc:2368] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:24.494637 9929 raft_consensus.cc:2393] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 4: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:24.521852 11769 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:24.495268 9934 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:24.496258 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:24.499614 10568 raft_consensus.cc:1240] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260430 07:59:24.527024 9933 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:24.505443 9932 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:24.483516 12441 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:24.527748 12493 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.527809 12441 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.530159 12441 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.516718 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1093) took 1723 ms. Trace:
I20260430 07:59:24.531167 9934 raft_consensus.cc:1275] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:24.530802 10005 rpcz_store.cc:276] 0430 07:59:22.793341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:22.794263 (+ 922us) service_pool.cc:225] Handling call
0430 07:59:22.794693 (+ 430us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:23.755881 (+961188us) spinlock_profiling.cc:243] Waited 3.88 ms on lock 0x7b18003cdea0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb568e2a85c 00007fb5661c8c0b 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e2a95 00007fb56927b440 00007fb56927bfdb 00007fb569286a2b
0430 07:59:23.756016 (+ 135us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:23.756045 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:23.772910 (+ 16865us) log.cc:844] Serialized 26 byte log entry
0430 07:59:24.072603 (+299693us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:24.073219 (+ 616us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:24.074513 (+ 1294us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:24.516442 (+441929us) raft_consensus.cc:1685] finished
0430 07:59:24.516463 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:24.516658 (+ 195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10469,"raft.queue_time_us":58386,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":64974,"spinlock_wait_cycles":8920960,"thread_start_us":385578,"threads_started":2,"wal-append.queue_time_us":668813}
I20260430 07:59:24.532986 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:24.321269 9854 rpcz_store.cc:276] 0430 07:59:23.287343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.288046 (+ 703us) service_pool.cc:225] Handling call
0430 07:59:23.302855 (+ 14809us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:23.303048 (+ 193us) raft_consensus.cc:1534] Early marking committed up to index 4
0430 07:59:23.474767 (+171719us) log.cc:844] Serialized 14 byte log entry
0430 07:59:24.300576 (+825809us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:24.300826 (+ 250us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:24.300879 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:24.300956 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:24.301094 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32705,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":10542,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":7180,"thread_start_us":976540,"threads_started":2,"wal-append.queue_time_us":841441}
W20260430 07:59:24.493592 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 187856us
I20260430 07:59:24.537544 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.538488 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.540885 9859 raft_consensus.cc:3055] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:24.540913 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.541749 9860 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:24.542809 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.543646 9861 raft_consensus.cc:2410] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
W20260430 07:59:24.544698 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 240947us
I20260430 07:59:24.541417 9859 raft_consensus.cc:740] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:24.525758 12275 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.548075 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:24.501104 12519 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:24.549360 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:24.549465 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.548952 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:24.555106 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:24.557611 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 358246us
I20260430 07:59:24.550482 9859 consensus_queue.cc:260] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.563478 9828 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:24.563613 9859 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:24.564265 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:24.548537 12547 log.cc:927] Time spent T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.176s user 0.003s sys 0.001s
I20260430 07:59:24.533535 12284 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:24.569000 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:24.571489 10010 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:24.572736 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.569098 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:24.549537 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:24.541447 12559 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:24.548015 12556 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:24.576123 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.576824 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:24.547844 12458 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:24.551955 12441 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:24.584617 10011 raft_consensus.cc:2393] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 7: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:24.561303 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:24.588116 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.588239 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:24.569352 12455 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:24.598627 9828 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:24.566118 12479 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:59:24.619855 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:24.643376 9760 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48094 (request call id 212) took 33832 ms (client timeout 15000 ms). Trace:
I20260430 07:59:24.621425 9977 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:24.579829 12529 consensus_queue.cc:1059] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.140s
I20260430 07:59:24.581166 12355 consensus_queue.cc:1059] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.202s
W20260430 07:59:24.719193 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 120653us
W20260430 07:59:24.744532 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:24.815114 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:24.680060 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:24.806212 12275 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.818902 9904 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:24.819996 9828 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:24.743150 9859 raft_consensus.cc:1275] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260430 07:59:24.837855 9859 pending_rounds.cc:85] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 3
I20260430 07:59:24.838299 9859 pending_rounds.cc:107] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260430 07:59:24.838711 9859 raft_consensus.cc:2889] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:24.587641 12556 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.812089 10560 raft_consensus.cc:2393] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:59:24.804162 12284 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:24.608314 12455 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:24.783640 12410 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:24.773699 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 270288us
I20260430 07:59:24.861840 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:24.804570 12566 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:24.783427 12423 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:24.850049 12550 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.368s user 0.000s sys 0.001s
I20260430 07:59:24.870151 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:24.582957 12559 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.849833 12284 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:24.888948 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 317375us
I20260430 07:59:24.890089 10011 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
W20260430 07:59:24.898284 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 308532us
I20260430 07:59:24.856455 12455 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:24.854679 12502 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:24.595904 12458 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.803983 12522 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.314s
I20260430 07:59:24.864439 12355 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:24.595707 12441 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.868633 12566 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:24.939513 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:24.943400 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 346154us
I20260430 07:59:24.721153 12479 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.898583 12517 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:24.870831 12410 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:24.945056 12355 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:24.892449 12423 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.067431 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:25.068712 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:25.060825 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:59:24.946955 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:24.980156 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:24.984913 12441 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.959045 10011 raft_consensus.cc:2410] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260430 07:59:24.946511 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 88999us
I20260430 07:59:24.994320 12586 consensus_queue.cc:1048] T 926617cd6f3149f6a70ccb92f4504659 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.169s
I20260430 07:59:24.985157 12284 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.958191 12256 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:25.156728 12256 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:25.157316 12256 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:25.159035 12423 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:24.976624 12444 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:25.161695 12424 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:25.162251 12444 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:24.644163 9760 rpcz_store.cc:269] 0430 07:58:50.752438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:58:50.758770 (+ 6332us) service_pool.cc:225] Handling call
0430 07:59:24.584319 (+33825549us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:59:15.597874 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:59:15.612002 (+ 14128us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:59:15.612053 (+ 51us) write_op.cc:435] Acquired schema lock
0430 07:59:15.612085 (+ 32us) tablet.cc:662] Decoding operations
0430 07:59:15.612995 (+ 910us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:59:15.613082 (+ 87us) write_op.cc:641] Partition lock acquired for write op
0430 07:59:15.613109 (+ 27us) tablet.cc:685] Acquiring locks for 6 operations
0430 07:59:15.613819 (+ 710us) tablet.cc:701] Row locks acquired
0430 07:59:15.613859 (+ 40us) write_op.cc:260] PREPARE: finished
0430 07:59:15.614079 (+ 220us) write_op.cc:270] Start()
0430 07:59:15.614221 (+ 142us) write_op.cc:276] Timestamp: P: 1777535955614010 usec, L: 0
0430 07:59:15.614259 (+ 38us) op_driver.cc:348] REPLICATION: starting
0430 07:59:15.614992 (+ 733us) log.cc:844] Serialized 2476 byte log entry
0430 07:59:16.419461 (+804469us) op_driver.cc:464] REPLICATION: finished
0430 07:59:16.967837 (+548376us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb5661c882c 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e5242 00007fb56ccc5360
0430 07:59:17.960661 (+992824us) write_op.cc:301] APPLY: starting
0430 07:59:17.960790 (+ 129us) tablet.cc:1367] starting BulkCheckPresence
0430 07:59:17.961037 (+ 247us) tablet.cc:1370] finished BulkCheckPresence
0430 07:59:17.961080 (+ 43us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:59:18.052844 (+ 91764us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:59:18.053151 (+ 307us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0430 07:59:18.053193 (+ 42us) write_op.cc:312] APPLY: finished
0430 07:59:18.053543 (+ 350us) log.cc:844] Serialized 52 byte log entry
0430 07:59:18.310043 (+256500us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:59:18.341410 (+ 31367us) write_op.cc:454] Released schema lock
0430 07:59:18.349755 (+ 8345us) write_op.cc:341] FINISH: Updating metrics
0430 07:59:18.359229 (+ 9474us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:59:18.359267 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":501050,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":1535162,"apply.run_cpu_time_us":11296,"apply.run_wall_time_us":398552,"num_ops":6,"prepare.queue_time_us":220637,"prepare.run_cpu_time_us":9178,"prepare.run_wall_time_us":111629,"raft.queue_time_us":186831,"raft.run_cpu_time_us":70319,"raft.run_wall_time_us":1503873,"replication_time_us":617928,"spinlock_wait_cycles":27082496,"thread_start_us":1729532,"threads_started":5,"wal-append.queue_time_us":556445,"wal-append.run_cpu_time_us":24631,"wal-append.run_wall_time_us":2261777}]]}
I20260430 07:59:25.162910 12444 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:25.162829 12424 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.003314 12555 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:24.967801 12503 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.578s
I20260430 07:59:25.166574 12555 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:24.985852 12536 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:25.173228 12536 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:25.173906 12536 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:25.175208 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:25.176272 10011 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:25.018363 12568 log.cc:927] Time spent T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.111s user 0.000s sys 0.001s
I20260430 07:59:25.179162 9934 raft_consensus.cc:1275] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260430 07:59:24.974530 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 70475us
I20260430 07:59:24.973450 12517 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.004220 12556 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:24.978888 12355 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:25.149956 9859 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 7
I20260430 07:59:24.961544 12566 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:25.151986 12410 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.152369 12458 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.196029 12458 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:25.196959 12458 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.199983 12458 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:24.977550 12523 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:25.201751 12523 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.204914 12479 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:24.958272 9977 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:25.206086 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1071) took 1524 ms. Trace:
I20260430 07:59:25.206583 9938 rpcz_store.cc:276] 0430 07:59:23.681876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.682054 (+ 178us) service_pool.cc:225] Handling call
0430 07:59:23.706047 (+ 23993us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:23.760195 (+ 54148us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:25.197995 (+1437800us) spinlock_profiling.cc:243] Waited 41.9 ms on lock 0x7b54003232d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:25.205856 (+ 7861us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96341248}
W20260430 07:59:25.207870 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 250425us
I20260430 07:59:24.958703 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:25.213639 12590 consensus_queue.cc:1048] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260430 07:59:25.299597 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:25.161151 12423 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:25.209255 9977 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:25.357869 12284 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:25.358815 12556 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:25.365196 12556 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.366817 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:25.368311 10005 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:25.369490 12284 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.373004 12284 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.377478 12526 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:25.359724 12523 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.378615 9903 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:25.387096 12479 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:25.388948 12595 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:25.390743 10005 raft_consensus.cc:1275] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:25.391741 12596 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:25.405857 12424 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.417918 12596 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.392556 12583 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:25.398308 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:25.403414 12458 raft_consensus.cc:2749] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260430 07:59:25.409649 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 53371.9us
I20260430 07:59:25.417948 12393 consensus_queue.cc:1059] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:25.402685 12347 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:25.506088 10568 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:25.663110 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 284732us
W20260430 07:59:25.441696 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 232616us
I20260430 07:59:25.484244 9859 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:25.637218 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1105) took 2713 ms. Trace:
I20260430 07:59:25.681602 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:25.439121 10010 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1739) took 1638 ms (client timeout 2172 ms). Trace:
I20260430 07:59:25.681463 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:25.683080 10010 rpcz_store.cc:269] 0430 07:59:23.740726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.755600 (+ 14874us) service_pool.cc:225] Handling call
0430 07:59:25.227645 (+1472045us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b54000f7dd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:25.379190 (+151545us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1873007488}
W20260430 07:59:25.683990 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:25.438448 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 1118) took 1252 ms. Trace:
I20260430 07:59:25.685135 10009 rpcz_store.cc:276] 0430 07:59:24.186151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:24.227035 (+ 40884us) service_pool.cc:225] Handling call
0430 07:59:25.226737 (+999702us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b54000ce254. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:25.438193 (+211456us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1163895040}
I20260430 07:59:25.689688 10006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1113) took 1384 ms. Trace:
W20260430 07:59:25.684617 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:25.690218 10006 rpcz_store.cc:276] 0430 07:59:24.183751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:24.186346 (+ 2595us) service_pool.cc:225] Handling call
0430 07:59:24.186801 (+ 455us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:24.507825 (+321024us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:25.223530 (+715705us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b540018d8d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:25.567750 (+344220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":736712064}
I20260430 07:59:25.638907 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:25.693907 10567 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54770 (request call id 370) took 2384 ms (client timeout 1546 ms). Trace:
W20260430 07:59:25.694440 10567 rpcz_store.cc:269] 0430 07:59:23.309944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.310662 (+ 718us) service_pool.cc:225] Handling call
0430 07:59:25.638450 (+2327788us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b58001f1798. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:25.693801 (+ 55351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31784320}
W20260430 07:59:25.699198 12355 consensus_meta.cc:287] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.512s user 0.013s sys 0.001s
I20260430 07:59:25.699983 12355 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.635978 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1077) took 1197 ms. Trace:
I20260430 07:59:25.710415 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1520) took 1276 ms. Trace:
I20260430 07:59:25.710867 9933 rpcz_store.cc:276] 0430 07:59:24.434245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:24.527719 (+ 93474us) service_pool.cc:225] Handling call
0430 07:59:24.528064 (+ 345us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:24.528424 (+ 360us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:24.528456 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:24.529071 (+ 615us) log.cc:844] Serialized 26 byte log entry
0430 07:59:24.856187 (+327116us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:25.489322 (+633135us) log.cc:844] Serialized 14 byte log entry
0430 07:59:25.709664 (+220342us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:25.709778 (+ 114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:25.709872 (+ 94us) raft_consensus.cc:1685] finished
0430 07:59:25.709884 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:25.710220 (+ 336us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":129135,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":57057,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":189,"thread_start_us":817674,"threads_started":2,"wal-append.queue_time_us":399787}
W20260430 07:59:25.507385 12525 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260430 07:59:25.421250 12596 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.593159 12556 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:25.685153 12555 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:25.711231 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 140172us
W20260430 07:59:25.717926 10011 consensus_meta.cc:287] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.541s user 0.011s sys 0.003s
I20260430 07:59:25.719735 10011 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:25.511960 12532 raft_consensus.cc:3055] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:25.650547 12559 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.419452 12517 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.716604 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1111) took 2152 ms. Trace:
I20260430 07:59:25.511132 12531 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:25.719916 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 78984.3us
I20260430 07:59:25.726689 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:25.727624 9977 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:25.705475 9932 rpcz_store.cc:276] 0430 07:59:24.438281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:24.537642 (+ 99361us) service_pool.cc:225] Handling call
0430 07:59:24.566726 (+ 29084us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:24.580735 (+ 14009us) raft_consensus.cc:1534] Early marking committed up to index 4
0430 07:59:24.857309 (+276574us) log.cc:844] Serialized 14 byte log entry
0430 07:59:25.210577 (+353268us) log.cc:844] Serialized 14 byte log entry
0430 07:59:25.211072 (+ 495us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:25.211173 (+ 101us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:25.211304 (+ 131us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:25.635421 (+424117us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b540027e7d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:25.635499 (+ 78us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:25.635748 (+ 249us) inbound_call.cc:177] Queueing success response
0430 07:59:25.692889 (+ 57141us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b18002a4c60. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661b3be6 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661e5a33 00007fb5661e7e01
Metrics: {"mutex_wait_us":236185,"prepare.queue_time_us":398733,"prepare.run_wall_time_us":59,"raft.queue_time_us":45764,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":161,"spinlock_wait_cycles":67156736,"thread_start_us":364753,"threads_started":3,"wal-append.queue_time_us":128577}
I20260430 07:59:25.725863 10007 rpcz_store.cc:276] 0430 07:59:23.344893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.345027 (+ 134us) service_pool.cc:225] Handling call
0430 07:59:23.345361 (+ 334us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:23.763914 (+418553us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:25.224430 (+1460516us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b54002df0d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:25.497067 (+272637us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":960736512}
I20260430 07:59:25.739029 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:25.420903 12423 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.717377 12583 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.422076 12595 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.721969 12532 raft_consensus.cc:740] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:25.700986 9759 catalog_manager.cc:5671] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:25.746675 9759 catalog_manager.cc:5671] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:25.747738 12301 raft_consensus.cc:3055] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:25.749024 12301 raft_consensus.cc:740] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:25.749960 9759 catalog_manager.cc:5671] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:25.758199 9759 catalog_manager.cc:5671] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index 2 to 3, term changed from 1 to 5, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to d66db415f0f0402fa431349afbf59888 (127.8.119.132), d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:25.753732 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:25.766623 9857 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:25.754005 10004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 302) took 2470 ms. Trace:
W20260430 07:59:25.755334 9930 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1083) took 1303 ms (client timeout 1686 ms). Trace:
W20260430 07:59:25.755856 12256 consensus_meta.cc:287] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.598s user 0.011s sys 0.003s
I20260430 07:59:25.751361 9903 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:25.759397 12444 consensus_meta.cc:287] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.596s user 0.010s sys 0.003s
I20260430 07:59:25.760818 12485 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:25.764257 12284 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:25.768625 9930 rpcz_store.cc:269] 0430 07:59:24.451858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:24.459242 (+ 7384us) service_pool.cc:225] Handling call
0430 07:59:25.754880 (+1295638us) spinlock_profiling.cc:243] Waited 47.9 ms on lock 0x7b540027f6d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:25.755118 (+ 238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":110269824}
I20260430 07:59:25.770218 12485 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.770465 12284 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.767803 10004 rpcz_store.cc:276] 0430 07:59:23.283128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.307592 (+ 24464us) service_pool.cc:225] Handling call
0430 07:59:23.307932 (+ 340us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:23.485357 (+177425us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:23.485406 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:23.485470 (+ 64us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:23.485535 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:25.753559 (+2268024us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54003c0854. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:25.753652 (+ 93us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:25.753845 (+ 193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":407277824}
I20260430 07:59:25.767242 11769 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:25.773231 12284 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.773802 11769 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.774194 12256 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.776168 12583 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.776348 11769 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.679791 10008 rpcz_store.cc:276] 0430 07:59:22.924141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:22.935301 (+ 11160us) service_pool.cc:225] Handling call
0430 07:59:22.941303 (+ 6002us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:23.285483 (+344180us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:23.285578 (+ 95us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:23.286053 (+ 475us) log.cc:844] Serialized 26 byte log entry
0430 07:59:24.095754 (+809701us) raft_consensus.cc:1645] Marking committed up to 2
0430 07:59:24.095812 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:25.636410 (+1540598us) spinlock_profiling.cc:243] Waited 55.9 ms on lock 0x7b54000cdd54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:25.636503 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:25.636610 (+ 107us) raft_consensus.cc:1685] finished
0430 07:59:25.636632 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:25.637009 (+ 377us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":316502,"raft.queue_time_us":332466,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":123116,"spinlock_wait_cycles":128665344,"thread_start_us":809243,"threads_started":1,"wal-append.queue_time_us":1054086,"wal-append.run_cpu_time_us":2385,"wal-append.run_wall_time_us":1152384}
I20260430 07:59:25.766942 9759 catalog_manager.cc:5671] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
W20260430 07:59:25.753085 12455 consensus_meta.cc:287] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.819s user 0.008s sys 0.006s
I20260430 07:59:25.777815 12476 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:25.791896 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:25.781934 12256 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:25.785585 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 70192.2us
I20260430 07:59:25.797150 9933 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:25.760938 12347 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.780793 12355 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.773264 12485 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.802268 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:25.793462 12531 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:25.805948 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:25.805145 12483 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:25.811193 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 90690.3us
I20260430 07:59:25.780365 12444 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.802827 9759 catalog_manager.cc:5671] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
W20260430 07:59:25.818172 12432 consensus_meta.cc:287] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 1.412s user 0.015s sys 0.010s
I20260430 07:59:25.818821 12432 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:25.820272 9828 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.744s (SENT) [suppressed 16 similar messages]
I20260430 07:59:25.826752 12422 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:25.827409 9929 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:25.827564 12422 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.805773 11769 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:25.833513 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:25.800707 12455 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.814782 12483 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.843950 9932 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:25.822690 12256 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:25.846210 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:25.829077 12432 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.815537 12523 consensus_queue.cc:1059] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20260430 07:59:25.824402 12583 raft_consensus.cc:476] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Not starting pre-election -- already a leader
I20260430 07:59:25.856451 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:25.855683 9903 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:25.848531 9759 catalog_manager.cc:5671] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 6, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 6 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:25.864751 12421 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:25.865619 12421 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:25.867003 12524 raft_consensus.cc:3055] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:25.867453 12524 raft_consensus.cc:740] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:25.868404 12524 consensus_queue.cc:260] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.869753 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:25.870010 12536 consensus_meta.cc:287] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.695s user 0.012s sys 0.002s
I20260430 07:59:25.871179 12301 consensus_queue.cc:260] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.871762 12536 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.874554 12536 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.841789 12422 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:25.879729 9930 raft_consensus.cc:1240] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 3. Current term is 4. Ops: []
I20260430 07:59:25.883450 9931 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:25.873028 12301 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:25.849614 12531 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:25.832811 12532 consensus_queue.cc:260] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.886745 12393 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:25.904098 9860 raft_consensus.cc:3055] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Stepping down as leader of term 5
W20260430 07:59:25.906605 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 75794.6us
I20260430 07:59:25.908314 9977 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:25.909720 12455 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.910914 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:25.911101 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:25.909503 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:25.912439 9860 raft_consensus.cc:740] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:25.901404 9759 catalog_manager.cc:5671] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:25.898731 12524 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:25.922657 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:25.925458 12369 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:25.957610 12347 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.959626 12393 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.926313 12423 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.897346 12517 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:25.926924 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:25.927299 9930 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:25.927299 9860 consensus_queue.cc:260] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.961647 9933 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:25.961962 9860 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:25.963399 12421 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.928182 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:25.965611 9863 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:25.928658 12483 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.929118 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:25.929764 12599 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:25.929839 12488 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:25.967779 12599 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.968394 12488 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.970577 12599 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:25.930644 12566 consensus_meta.cc:287] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.704s user 0.013s sys 0.002s
I20260430 07:59:25.933964 12444 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:25.939118 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:25.945926 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:25.973671 12566 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.952437 10009 raft_consensus.cc:1240] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260430 07:59:25.974699 10004 raft_consensus.cc:1240] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:25.975481 9861 raft_consensus.cc:1240] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:25.976795 12556 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:25.953167 10011 raft_consensus.cc:1275] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260430 07:59:25.977036 12531 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.978588 9860 raft_consensus.cc:1275] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260430 07:59:25.981556 9860 pending_rounds.cc:85] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 3
I20260430 07:59:25.985875 9860 pending_rounds.cc:107] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260430 07:59:25.979938 12531 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:25.979931 12562 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:25.992692 9860 raft_consensus.cc:2889] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:25.992637 12562 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:25.994042 12393 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.889353 12256 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:25.976394 12566 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:25.954869 12595 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:25.991732 12453 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:25.952859 9938 raft_consensus.cc:1240] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260430 07:59:25.990060 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 66787us
I20260430 07:59:26.002219 9977 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:26.003485 12531 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:25.996344 12562 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:26.005210 9977 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:25.985707 12590 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:26.011577 12590 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:26.014554 12590 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:26.012063 12421 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:26.012163 10010 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260430 07:59:26.016865 9935 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:26.020831 9935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1068) took 2510 ms (client timeout 2007 ms). Trace:
I20260430 07:59:26.026268 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:26.029337 10010 raft_consensus.cc:1275] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:26.031266 12488 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:26.037451 9935 rpcz_store.cc:269] 0430 07:59:23.510646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.515763 (+ 5117us) service_pool.cc:225] Handling call
0430 07:59:26.020464 (+2504701us) spinlock_profiling.cc:243] Waited 2.45 s on lock 0x7b5400348d54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:26.020690 (+ 226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5624155008}
W20260430 07:59:26.026866 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 174651us
W20260430 07:59:26.035635 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 116895us
I20260430 07:59:26.038793 9860 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: []
W20260430 07:59:26.034523 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 137767us
I20260430 07:59:26.040449 9930 raft_consensus.cc:1275] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:26.053951 12532 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:26.032325 9759 catalog_manager.cc:5671] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index 2 to 3, term changed from 1 to 5, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to d66db415f0f0402fa431349afbf59888 (127.8.119.132), d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
W20260430 07:59:26.040076 9931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1066) took 2668 ms (client timeout 1744 ms). Trace:
I20260430 07:59:26.058003 9933 raft_consensus.cc:2468] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:26.064072 12421 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:26.064365 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 59295us
I20260430 07:59:26.065455 9977 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:26.066499 12531 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:26.067441 12256 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:26.066958 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:26.062487 9931 rpcz_store.cc:269] 0430 07:59:23.371996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.372348 (+ 352us) service_pool.cc:225] Handling call
0430 07:59:25.748307 (+2375959us) spinlock_profiling.cc:243] Waited 98.3 ms on lock 0x7b54003b1120. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5691fb1a0 00007fb5692871f8 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:26.039805 (+291498us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400338bd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:26.039996 (+ 191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1397777152}
I20260430 07:59:26.081233 12488 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:26.081979 12488 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:26.082397 9903 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:26.082252 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.079422 12556 consensus_queue.cc:1048] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260430 07:59:26.103747 10010 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:26.104877 10568 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:26.084740 9759 catalog_manager.cc:5671] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 5, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 5 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:26.110769 9936 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:26.106599 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.127138 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:26.128959 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 43660.5us
I20260430 07:59:26.133601 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:26.137961 10568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54770 (request call id 375) took 1183 ms. Trace:
I20260430 07:59:26.143824 10568 rpcz_store.cc:276] 0430 07:59:24.946524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:24.946652 (+ 128us) service_pool.cc:225] Handling call
0430 07:59:26.105771 (+1159119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:26.104465 11769 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:26.148018 9903 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:26.103422 12476 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:26.126870 12410 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:26.159716 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1081) took 1708 ms. Trace:
I20260430 07:59:26.133886 12536 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:26.133457 12531 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:26.170897 10006 raft_consensus.cc:1275] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260430 07:59:26.172156 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:26.184422 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 73895.2us
W20260430 07:59:26.189775 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:26.195076 10010 raft_consensus.cc:1240] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:26.175269 9759 catalog_manager.cc:5671] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:26.161412 9934 rpcz_store.cc:276] 0430 07:59:24.451240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:25.180710 (+729470us) service_pool.cc:225] Handling call
0430 07:59:25.356333 (+175623us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:25.378513 (+ 22180us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:25.664056 (+285543us) log.cc:844] Serialized 14 byte log entry
0430 07:59:26.150703 (+486647us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:26.150762 (+ 59us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:26.150814 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:26.151835 (+ 1021us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:26.158582 (+ 6747us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15616,"prepare.queue_time_us":295928,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":22537,"thread_start_us":401587,"threads_started":1}
I20260430 07:59:26.210722 9759 catalog_manager.cc:5671] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 reported cstate change: config changed from index 2 to 4, term changed from 2 to 4, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to d66db415f0f0402fa431349afbf59888 (127.8.119.132), d66db415f0f0402fa431349afbf59888 (127.8.119.132) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
W20260430 07:59:26.219904 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 108841us
I20260430 07:59:26.227541 9903 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:26.188212 12590 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:26.227694 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:26.252696 9936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1069) took 2686 ms (client timeout 1836 ms). Trace:
W20260430 07:59:26.253329 9936 rpcz_store.cc:269] 0430 07:59:23.566104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:23.568205 (+ 2101us) service_pool.cc:225] Handling call
0430 07:59:26.238708 (+2670503us) spinlock_profiling.cc:243] Waited 2.39 s on lock 0x7b54001ee454. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:26.252605 (+ 13897us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5501251456}
I20260430 07:59:26.173445 12410 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:26.263996 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:26.262104 11769 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:26.287304 11769 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:26.287866 11769 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:26.293452 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 72350.9us
I20260430 07:59:26.229746 12536 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:26.228288 12531 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:26.301939 9759 catalog_manager.cc:5671] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 3, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 3 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
I20260430 07:59:26.272521 12518 consensus_queue.cc:1059] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
W20260430 07:59:26.325943 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 101833us
I20260430 07:59:26.323319 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:26.268491 12479 consensus_queue.cc:1059] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260430 07:59:26.329983 10011 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:26.341879 12518 raft_consensus.cc:3055] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:26.342437 12518 raft_consensus.cc:740] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:26.343467 12518 consensus_queue.cc:260] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:26.314883 12599 consensus_queue.cc:1048] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.033s
W20260430 07:59:26.346210 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:26.345062 12518 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:26.327737 12488 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:26.355943 12556 consensus_queue.cc:1048] T 725a25b9eaa049c3ba8535dbe1100561 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:26.365479 9759 catalog_manager.cc:5671] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 5, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 5 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:26.368712 10006 raft_consensus.cc:1275] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:26.352700 12479 consensus_queue.cc:1243] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer 27f97a330f374849be58a168318c1ec6 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260430 07:59:26.363919 12421 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:26.384742 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 62585.9us
I20260430 07:59:26.386245 9759 catalog_manager.cc:5671] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
W20260430 07:59:26.389082 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:26.400058 9903 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:59:26.413228 12602 log.cc:927] Time spent T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.182s user 0.001s sys 0.000s
W20260430 07:59:26.414227 12601 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260430 07:59:26.416622 9759 catalog_manager.cc:5671] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:26.419916 9977 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:26.466421 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:26.409710 12451 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.049s sys 0.078s
I20260430 07:59:26.465986 12432 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:26.468070 12531 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:26.474972 12549 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:26.482796 9759 catalog_manager.cc:5671] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 6, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 6 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:26.480729 12421 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260430 07:59:26.519512 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 125137us
I20260430 07:59:26.488515 12536 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:26.549779 12603 log.cc:927] Time spent T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20260430 07:59:26.559942 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:26.561663 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 150899us
W20260430 07:59:26.559245 12451 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.871s user 0.050s sys 0.079s
I20260430 07:59:26.569701 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:26.573624 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:26.566534 12432 raft_consensus.cc:697] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
W20260430 07:59:26.567394 12606 log.cc:927] Time spent T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20260430 07:59:26.585675 9863 raft_consensus.cc:2410] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260430 07:59:26.567070 12421 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:26.593616 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:26.582686 9759 catalog_manager.cc:5671] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 4 to 5, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 5 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:26.602203 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:26.603967 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:26.581617 12549 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:26.605849 12604 log.cc:927] Time spent T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
W20260430 07:59:26.606020 12608 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260430 07:59:26.607174 10003 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1117) took 2418 ms. Trace:
I20260430 07:59:26.607700 10003 rpcz_store.cc:276] 0430 07:59:24.185744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:24.227632 (+ 41888us) service_pool.cc:225] Handling call
0430 07:59:24.229607 (+ 1975us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:24.284627 (+ 55020us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:24.284659 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:24.285201 (+ 542us) log.cc:844] Serialized 26 byte log entry
0430 07:59:25.756260 (+1471059us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b180039d960. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661c8587 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e5242 00007fb568e1c5ab 00007fb568e220c3 00007fb568e22471 00007fb56923edef
0430 07:59:26.208463 (+452203us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:26.224178 (+ 15715us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:26.284935 (+ 60757us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:26.599045 (+314110us) raft_consensus.cc:1685] finished
0430 07:59:26.599067 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:26.604012 (+ 4945us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":76016,"raft.queue_time_us":83840,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":3251,"spinlock_wait_cycles":5579520,"thread_start_us":371924,"threads_started":1,"wal-append.queue_time_us":2043412}
W20260430 07:59:26.621066 9828 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.015s sys 0.071s
I20260430 07:59:26.620769 12432 consensus_queue.cc:237] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:26.621765 9828 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.801s user 0.018s sys 0.071s
I20260430 07:59:26.596959 12421 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:26.622443 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 802187us
W20260430 07:59:26.626700 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:26.627576 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:26.628966 12532 consensus_meta.cc:287] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.550s user 0.008s sys 0.002s
I20260430 07:59:26.624871 9977 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:26.631592 12590 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:26.621822 12607 log.cc:927] Time spent T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20260430 07:59:26.638641 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:26.638818 9759 catalog_manager.cc:5671] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 6, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 6 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:26.641249 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:26.644232 9862 raft_consensus.cc:2410] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260430 07:59:26.646705 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:26.647490 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:26.629235 12488 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:26.646593 12483 raft_consensus.cc:2749] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:26.696535 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:26.697932 9759 catalog_manager.cc:5671] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 6, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 6 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:26.696312 9935 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:26.700845 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 57694.9us
I20260430 07:59:26.691444 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:26.701692 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 304037us
I20260430 07:59:26.701817 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:26.703869 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:26.692540 9904 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:26.708963 9863 raft_consensus.cc:2410] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260430 07:59:26.709312 9759 catalog_manager.cc:5671] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 5, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 5 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
I20260430 07:59:26.712347 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:26.708117 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.718107 10006 raft_consensus.cc:1275] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260430 07:59:26.720009 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 95230.8us
I20260430 07:59:26.723112 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:26.727196 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:26.732038 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 91512.5us
I20260430 07:59:26.735536 10004 raft_consensus.cc:2410] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260430 07:59:26.738001 9904 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:26.714296 12301 consensus_meta.cc:287] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.825s user 0.024s sys 0.028s
I20260430 07:59:26.696327 12488 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:26.746364 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.746784 9857 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:26.748253 9977 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:26.739781 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.755545 10003 raft_consensus.cc:2393] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:26.758599 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:26.725581 12453 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:26.759599 9934 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:26.764797 9862 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 8. Current term is 9. Ops: []
I20260430 07:59:26.757586 9759 catalog_manager.cc:5671] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 3 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:26.714893 12590 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:26.771075 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.776250 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:26.778208 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:26.748051 12536 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election won for term 7
W20260430 07:59:26.795955 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:26.722179 12532 consensus_queue.cc:1048] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260430 07:59:26.798031 9759 catalog_manager.cc:5671] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 5, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 5 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:26.799659 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.800581 9828 leader_election.cc:400] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:26.801031 9828 leader_election.cc:403] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260430 07:59:26.777740 12369 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:26.809199 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.778568 12256 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:26.813825 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:26.816819 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 85852.2us
I20260430 07:59:26.820459 12450 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:26.821241 9977 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:26.775256 12453 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:26.831610 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 62156.8us
I20260430 07:59:26.803551 12536 raft_consensus.cc:697] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:26.782976 12556 consensus_queue.cc:1059] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260430 07:59:26.810195 12476 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:26.817601 12256 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:26.814947 12369 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:26.830400 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.851357 9935 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:26.862406 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.849185 12556 consensus_queue.cc:1243] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer 27f97a330f374849be58a168318c1ec6 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260430 07:59:26.849395 12256 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:26.857916 12369 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:26.882180 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.881824 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.882529 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:26.877545 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.884435 9903 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:26.886219 9855 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260430 07:59:26.891397 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.870823 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:26.876983 9828 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:26.860816 12483 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260430 07:59:26.885519 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 64361.2us
I20260430 07:59:26.907797 10004 raft_consensus.cc:2393] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:26.885792 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.888340 9932 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:26.885664 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.903745 12488 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:26.911553 9977 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:26.911684 10007 raft_consensus.cc:2410] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260430 07:59:26.912889 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.913419 9977 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:26.913686 10008 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:26.914686 12556 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:26.915032 10009 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:26.915655 12479 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:26.913417 10010 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 9
I20260430 07:59:26.911787 12483 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:26.917310 12483 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:26.916369 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:26.921536 10005 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:26.911686 10003 raft_consensus.cc:2410] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260430 07:59:26.924939 12556 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:26.925115 12453 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:26.926344 10010 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 9.
W20260430 07:59:26.927445 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:26.877578 12301 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.053s
I20260430 07:59:26.915899 10006 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:26.937978 12556 consensus_queue.cc:1059] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:59:26.938897 12556 consensus_queue.cc:1243] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer 27f97a330f374849be58a168318c1ec6 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260430 07:59:26.952754 12410 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:26.961706 12549 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:26.966215 12478 raft_consensus.cc:3055] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:26.969784 12422 raft_consensus.cc:3055] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 8 LEADER]: Stepping down as leader of term 8
W20260430 07:59:26.982442 10012 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 1114) took 2797 ms (client timeout 1515 ms). Trace:
W20260430 07:59:26.992802 12518 consensus_meta.cc:287] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.627s user 0.009s sys 0.000s
I20260430 07:59:26.980996 12483 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.063151 12483 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:26.889526 12451 consensus_queue.cc:1059] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.124s
I20260430 07:59:27.004997 12590 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:26.865295 9759 catalog_manager.cc:5671] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
W20260430 07:59:26.879196 12605 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260430 07:59:26.989084 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1097) took 1067 ms. Trace:
W20260430 07:59:27.075505 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 372599us
W20260430 07:59:27.026654 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 160050us
W20260430 07:59:27.053857 10012 rpcz_store.cc:269] 0430 07:59:24.184248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:24.184401 (+ 153us) service_pool.cc:225] Handling call
0430 07:59:26.981526 (+2797125us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x7b540037fad4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:26.981954 (+ 428us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4504952064}
W20260430 07:59:27.053293 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 168872us
I20260430 07:59:27.052809 12422 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:59:26.997217 11769 consensus_meta.cc:287] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.709s user 0.008s sys 0.004s
I20260430 07:59:26.920084 12421 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:26.849212 12476 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:27.041350 12599 raft_consensus.cc:2749] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:27.052443 12566 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:27.075703 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:27.042579 12478 raft_consensus.cc:740] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:27.039495 9857 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:27.080041 9903 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:26.877888 12536 consensus_queue.cc:237] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:27.073951 9930 rpcz_store.cc:276] 0430 07:59:26.054567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:26.069347 (+ 14780us) service_pool.cc:225] Handling call
0430 07:59:26.252717 (+183370us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:26.308841 (+ 56124us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:26.308879 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:26.312114 (+ 3235us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:26.312160 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:26.986954 (+674794us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b54001ee454. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:26.987064 (+ 110us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:26.988810 (+ 1746us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54001ee450. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:26.988995 (+ 185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8532,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":6042,"spinlock_wait_cycles":435393024}
I20260430 07:59:27.082183 9759 catalog_manager.cc:5671] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 4, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 4 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:27.091743 9828 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:27.128881 12451 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:27.129742 9828 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:27.128794 12478 consensus_queue.cc:260] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.135701 12478 raft_consensus.cc:3060] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:27.134898 9759 catalog_manager.cc:5671] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 4, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 4 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:27.136507 12369 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:27.135540 12301 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:59:27.136927 9759 catalog_manager.cc:5671] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:27.146095 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1735) took 1242 ms. Trace:
I20260430 07:59:27.146775 9857 rpcz_store.cc:276] 0430 07:59:25.896084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:25.899753 (+ 3669us) service_pool.cc:225] Handling call
0430 07:59:27.137799 (+1238046us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b540010ca54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:27.137960 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1920107904}
I20260430 07:59:27.090216 12566 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:27.150823 9759 catalog_manager.cc:5671] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:27.129515 12451 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:27.089078 11769 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.158154 9978 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:27.132391 12549 raft_consensus.cc:3055] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:27.125241 12590 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:27.174091 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 436090us
I20260430 07:59:27.170307 9759 catalog_manager.cc:5671] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 2 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
I20260430 07:59:27.146247 12532 raft_consensus.cc:2804] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:27.183578 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1720) took 1273 ms. Trace:
I20260430 07:59:27.155390 12566 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:27.156396 12613 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:27.171478 12451 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:27.200768 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 120836us
I20260430 07:59:27.171964 12549 raft_consensus.cc:740] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:27.186818 9854 rpcz_store.cc:276] 0430 07:59:25.904579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:25.916694 (+ 12115us) service_pool.cc:225] Handling call
0430 07:59:25.938822 (+ 22128us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:25.939370 (+ 548us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:25.939416 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:25.940167 (+ 751us) log.cc:844] Serialized 26 byte log entry
0430 07:59:26.157584 (+217417us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:26.157628 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:26.157702 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:27.176008 (+1018306us) raft_consensus.cc:1685] finished
0430 07:59:27.176031 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:27.176160 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14515,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":214,"thread_start_us":216840,"threads_started":1,"wal-append.queue_time_us":796877}
W20260430 07:59:27.202845 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 365234us
I20260430 07:59:27.207206 9932 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
W20260430 07:59:27.207540 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 57198.3us
I20260430 07:59:27.198549 9759 catalog_manager.cc:5671] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 2 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:27.208772 9759 catalog_manager.cc:5671] T 38bcd78c11574f1a9e016f9bb9365ca5 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 3 to 4, leader changed from 48e81934d52a4e93be5171ceea193032 (127.8.119.131) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 4 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
I20260430 07:59:27.210572 9759 catalog_manager.cc:5671] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 3, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 3 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
I20260430 07:59:27.212164 9759 catalog_manager.cc:5671] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 2, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 2 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
I20260430 07:59:27.195194 12532 raft_consensus.cc:697] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:27.199023 12479 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:27.152509 12422 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.221542 12476 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:27.222635 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:27.250223 10006 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "77e19ab653f540339c1c613fad1451b5"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:27.251005 10006 raft_consensus.cc:606] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Received request to transfer leadership to TS 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:27.251703 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.252219 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.251729 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.253623 10004 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:27.252969 10010 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:27.225766 12549 consensus_queue.cc:260] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.255600 12549 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:27.256479 12421 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:27.251358 10006 raft_consensus.cc:614] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:27.230746 12369 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:27.214716 12613 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:27.231921 11769 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:27.237506 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 48351.9us
I20260430 07:59:27.259876 9904 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:27.239549 9903 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.240263 12488 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:27.260619 12613 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:27.246197 12479 raft_consensus.cc:697] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:27.262853 12532 consensus_queue.cc:237] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.263098 12479 consensus_queue.cc:237] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:27.268362 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.269217 10006 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:27.269600 12422 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:27.269853 10006 raft_consensus.cc:606] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.270272 10006 raft_consensus.cc:614] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:27.270927 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.273407 10010 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:27.274104 10010 raft_consensus.cc:606] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.274629 10010 raft_consensus.cc:614] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:27.274024 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.275651 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.277661 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.277774 10010 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:27.278388 10010 raft_consensus.cc:606] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.278786 10010 raft_consensus.cc:614] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:27.245596 9759 catalog_manager.cc:5671] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 1 to 5, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to 48e81934d52a4e93be5171ceea193032 (127.8.119.131). New cstate: current_term: 5 leader_uuid: "48e81934d52a4e93be5171ceea193032" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: HEALTHY } } }
W20260430 07:59:27.289618 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 376242us
W20260430 07:59:27.302040 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 79757.1us
I20260430 07:59:27.302532 9759 catalog_manager.cc:5671] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 reported cstate change: term changed from 2 to 4, leader changed from 27f97a330f374849be58a168318c1ec6 (127.8.119.129) to 77e19ab653f540339c1c613fad1451b5 (127.8.119.130). New cstate: current_term: 4 leader_uuid: "77e19ab653f540339c1c613fad1451b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } }
W20260430 07:59:27.302284 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 54547.7us
W20260430 07:59:27.310148 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:27.295820 12256 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.312232 12256 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:27.311995 12451 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.307644 12450 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:27.310204 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:27.314512 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 75052.3us
I20260430 07:59:27.302501 12369 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:27.295264 12524 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:27.302352 12488 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:27.322168 9903 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.322607 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:27.325300 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.326195 10003 raft_consensus.cc:2393] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 9: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:27.324851 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.330999 12619 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:27.331681 12619 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.319449 12369 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:27.333666 12369 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:27.334609 12618 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:27.337797 9903 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.338665 12599 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:27.339818 9903 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.340970 12524 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.341579 12451 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:27.342511 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.343178 12524 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:27.343483 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:27.344420 12476 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:27.345742 9903 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.348728 9857 raft_consensus.cc:1240] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:27.307245 12566 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.349563 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:27.350688 9931 raft_consensus.cc:2393] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 9: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:27.350936 12478 consensus_queue.cc:1059] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:27.352257 12531 raft_consensus.cc:3055] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:27.352737 12531 raft_consensus.cc:740] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:27.354174 9977 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.353996 12531 consensus_queue.cc:260] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.355341 12531 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:27.358546 12451 consensus_queue.cc:1059] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:27.317379 12453 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260430 07:59:27.317520 11769 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260430 07:59:27.360350 12453 consensus_queue.cc:1048] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:59:27.360718 12566 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:27.362721 9903 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.314227 12301 consensus_queue.cc:1059] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260430 07:59:27.313495 12450 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.369570 12619 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:27.370577 12450 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:27.370720 12623 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:27.366804 12476 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:27.377810 12623 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:27.378367 12623 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:27.382159 12613 consensus_queue.cc:1059] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260430 07:59:27.383656 12447 raft_consensus.cc:3055] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:27.384069 12447 raft_consensus.cc:740] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:27.388904 9903 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.384954 12447 consensus_queue.cc:260] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:27.392076 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:27.394438 10568 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:27.399734 9903 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.403417 9903 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.407953 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:27.409252 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:27.411604 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:27.392100 12488 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:27.403064 12599 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:27.414057 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:27.417974 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:27.412984 12447 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:27.418900 9861 raft_consensus.cc:2393] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:27.420758 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:27.423642 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 64861.8us
I20260430 07:59:27.420835 9937 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236"
dest_uuid: "77e19ab653f540339c1c613fad1451b5"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:52704
I20260430 07:59:27.427194 9937 raft_consensus.cc:606] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.427707 9937 raft_consensus.cc:614] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:27.401156 12450 raft_consensus.cc:3055] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:27.433820 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:27.432345 9828 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:27.423307 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:27.437474 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:27.442485 9977 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:27.403301 12536 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 07:59:27.448695 10568 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 9.
I20260430 07:59:27.449470 12432 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:27.450459 12478 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:27.458948 12590 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:27.468747 12478 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.470984 12478 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:27.438689 12451 consensus_queue.cc:1048] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260430 07:59:27.458787 12422 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:27.450963 12301 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:27.451967 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:27.452679 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 49961.9us
I20260430 07:59:27.474593 9903 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.475839 9903 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.435325 12450 raft_consensus.cc:740] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:27.385499 12369 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:27.454568 9977 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:27.479451 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.480818 12563 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:27.480926 12619 raft_consensus.cc:3055] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:27.481433 12432 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.481881 12619 raft_consensus.cc:740] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:27.482802 12619 consensus_queue.cc:260] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.483633 12432 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:27.484257 12619 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:27.459059 12599 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:27.485648 12599 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.488880 12599 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:27.466614 12518 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:27.513447 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.491245 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:27.514364 9863 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:27.515407 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.492481 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:27.498276 12301 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:27.520514 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:27.521211 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:27.521864 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.523577 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:27.499158 12631 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:27.501497 12451 consensus_queue.cc:1048] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20260430 07:59:27.502712 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:27.528815 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.530851 9929 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:27.522688 10005 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:27.502657 12450 consensus_queue.cc:260] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.522806 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:27.532608 9938 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:27.533026 12450 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:27.533836 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.533555 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:27.535774 9935 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:27.536021 9828 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:27.511083 9903 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.536857 10010 raft_consensus.cc:1275] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:27.533586 12422 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:27.537521 12410 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:27.537915 12410 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:27.537914 12422 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.538537 12410 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:27.513927 12518 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.540199 12422 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:27.517690 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:27.541536 12631 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:27.513737 12476 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:27.520147 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:27.542770 10004 raft_consensus.cc:2410] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260430 07:59:27.517750 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 63284.9us
I20260430 07:59:27.543373 9828 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:27.543987 9977 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:27.545328 9977 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.546077 12631 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:27.546259 12478 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:27.517551 11769 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:27.546974 12478 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:27.547389 12478 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:27.547729 12478 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:27.536168 9931 raft_consensus.cc:1240] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:27.538455 10008 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:27.537329 9903 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.508606 10011 raft_consensus.cc:1240] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:27.551784 9903 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.552191 10008 raft_consensus.cc:606] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.552224 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:27.499022 12532 consensus_queue.cc:1059] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:27.560415 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:27.554425 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.557122 9935 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:27.565697 9936 raft_consensus.cc:1240] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:27.549587 12631 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:27.566152 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:27.566419 12631 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.567543 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.568483 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.568652 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:27.568871 12631 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:27.569094 10003 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:27.569475 10012 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
W20260430 07:59:27.601814 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 50103.9us
I20260430 07:59:27.602660 9903 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.603760 9903 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.569749 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.567574 9828 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:27.562420 10010 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:27.604470 10004 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:27.608052 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 52454.7us
I20260430 07:59:27.514810 12479 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:27.620121 12532 raft_consensus.cc:3055] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:27.620666 12532 raft_consensus.cc:740] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:27.621477 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 53988.9us
I20260430 07:59:27.621616 12631 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:27.623803 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260430 07:59:27.623636 12622 consensus_queue.cc:1059] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260430 07:59:27.626123 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:59:27.626539 12536 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:27.627718 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:59:27.627826 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:27.628647 9932 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:27.626652 12622 raft_consensus.cc:3055] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:27.632309 12622 raft_consensus.cc:740] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:27.639387 9937 raft_consensus.cc:1275] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260430 07:59:27.571686 11769 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260430 07:59:27.641592 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:27.641685 9977 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.643236 10008 raft_consensus.cc:1240] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260430 07:59:27.644008 10011 raft_consensus.cc:1240] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260430 07:59:27.612404 12599 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:27.645444 11769 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:27.645807 12599 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:27.645977 11769 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:27.646234 12599 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:27.646364 9903 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:27.647372 9903 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.648051 12617 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election won for term 9
I20260430 07:59:27.648538 12617 raft_consensus.cc:697] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 9 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:27.648481 9903 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.614722 12476 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:27.649647 12617 consensus_queue.cc:237] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:27.651269 12618 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:27.651687 12618 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:27.652017 12618 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:27.655339 9903 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.656963 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.658604 9903 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.659520 10012 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:27.661873 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:27.672184 9903 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:59:27.672435 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 56020.8us
I20260430 07:59:27.656543 12432 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:27.671780 12590 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:27.678252 10007 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260430 07:59:27.680902 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:27.679522 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:27.679256 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:27.682118 12590 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:27.704874 12590 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:27.682410 12432 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:27.695720 12599 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.663902 12422 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:27.711858 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:27.664819 12536 raft_consensus.cc:2764] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260430 07:59:27.690891 12369 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:27.715441 9937 raft_consensus.cc:2368] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8.
I20260430 07:59:27.718310 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:27.720906 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:27.718398 9977 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.724053 12422 raft_consensus.cc:697] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:27.724643 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.725310 10012 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:27.621573 12532 consensus_queue.cc:260] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.734869 12599 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:27.663548 12623 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.735087 10560 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:27.744513 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.747718 10004 raft_consensus.cc:1275] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:27.638538 12622 consensus_queue.cc:260] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:27.749991 10012 raft_consensus.cc:2393] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 8: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:27.751377 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:27.725119 12422 consensus_queue.cc:237] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.729362 12476 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:27.758224 10009 raft_consensus.cc:1240] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:27.763247 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:27.763916 10012 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:27.749625 12483 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:27.770931 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:27.782168 12636 log.cc:927] Time spent T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.089s user 0.003s sys 0.000s
W20260430 07:59:27.784250 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:27.770073 9828 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:59:27.792482 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:27.761535 12532 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:27.794274 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:27.800424 9933 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:27.796351 9828 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:27.796744 9938 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
I20260430 07:59:27.801086 9931 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260430 07:59:27.801874 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:27.802636 9938 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:27.765831 12518 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:27.807690 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 139643us
I20260430 07:59:27.808948 9903 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:27.811646 9903 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:27.813831 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:27.814585 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:27.816867 9903 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:27.819428 12622 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:27.815423 9828 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:27.827437 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:27.819753 12476 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:27.832222 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:27.836606 9828 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:27.835630 9904 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:27.804533 12549 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:27.837181 9855 raft_consensus.cc:1275] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260430 07:59:27.839145 9828 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:27.845041 9855 pending_rounds.cc:85] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 4
I20260430 07:59:27.845474 9855 pending_rounds.cc:107] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260430 07:59:27.845774 9855 raft_consensus.cc:2889] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:27.839313 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:27.847077 9938 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:27.805683 12483 raft_consensus.cc:697] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:27.855317 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:27.812935 12619 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:27.819991 12536 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:27.871415 9934 raft_consensus.cc:1275] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:27.883076 9934 raft_consensus.cc:1275] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260430 07:59:27.885218 12635 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.207s user 0.002s sys 0.000s
W20260430 07:59:27.894493 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 77672us
I20260430 07:59:27.896231 9903 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:27.841331 12451 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:27.812729 12566 consensus_queue.cc:1048] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.062s
I20260430 07:59:27.849148 12524 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:27.912266 9930 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260430 07:59:27.918021 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:27.870144 12453 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:27.869841 12619 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:27.831080 11769 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:27.931901 9903 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:27.955037 9933 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
W20260430 07:59:27.972340 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:27.905820 12488 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:27.978439 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:27.964485 10006 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:27.899026 12640 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:27.911705 12451 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:27.962647 12641 log.cc:927] Time spent T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.176s user 0.001s sys 0.000s
I20260430 07:59:27.927784 12524 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:27.997723 9903 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.899305 12549 consensus_queue.cc:1059] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260430 07:59:27.961979 12432 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260430 07:59:28.003345 9932 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 7: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:27.970679 12619 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:28.010253 9903 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:27.950763 12453 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:27.994918 12531 consensus_meta.cc:287] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.639s user 0.011s sys 0.003s
I20260430 07:59:28.013868 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:27.985529 12369 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:59:28.022967 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 161369us
I20260430 07:59:27.994717 12640 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:28.000212 12451 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:27.928099 12536 consensus_queue.cc:1048] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
W20260430 07:59:28.017997 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 220606us
I20260430 07:59:28.034574 10012 raft_consensus.cc:1275] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:27.995576 12623 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:28.004806 12524 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:28.040777 9861 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260430 07:59:27.983032 12590 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.008896 12599 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:28.018111 12453 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:28.069972 10568 raft_consensus.cc:1275] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260430 07:59:28.055478 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 195349us
I20260430 07:59:28.087697 9934 raft_consensus.cc:1240] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260430 07:59:28.096328 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 86109.5us
I20260430 07:59:28.017097 12410 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.030390 12640 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:28.042328 12369 raft_consensus.cc:697] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:28.069061 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:28.053958 12566 raft_consensus.cc:3055] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:28.113001 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:28.117981 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:28.127418 9935 raft_consensus.cc:1240] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
W20260430 07:59:28.115923 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 75627.4us
I20260430 07:59:28.021880 12483 consensus_queue.cc:237] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.140129 10568 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:28.114575 12658 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:28.155169 12369 consensus_queue.cc:237] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:28.166338 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:28.173583 11769 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.172441 12524 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:28.175690 12478 consensus_meta.cc:287] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.628s user 0.009s sys 0.000s
I20260430 07:59:28.176175 12478 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.178367 12410 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.118444 12566 raft_consensus.cc:740] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:28.182394 12566 consensus_queue.cc:260] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.180794 12524 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.184226 12566 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:28.187430 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 46693.6us
W20260430 07:59:28.188858 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.195940 9861 raft_consensus.cc:1240] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: []
W20260430 07:59:28.203953 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 90943.6us
I20260430 07:59:28.209826 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:28.212536 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.214893 12478 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:28.216305 10005 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:28.116453 12450 consensus_queue.cc:1059] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260430 07:59:28.225884 12450 raft_consensus.cc:3055] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:28.226315 12450 raft_consensus.cc:740] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:28.227198 12450 consensus_queue.cc:260] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.228492 12450 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:28.247730 9863 raft_consensus.cc:1240] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 8. Current term is 9. Ops: []
W20260430 07:59:28.219264 12618 consensus_meta.cc:287] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.554s user 0.000s sys 0.011s
I20260430 07:59:28.254734 10568 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
W20260430 07:59:28.254820 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 180516us
I20260430 07:59:28.256359 9828 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:28.139578 12301 consensus_queue.cc:1059] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.110s
I20260430 07:59:28.262161 12631 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:28.268400 12631 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:28.268942 12631 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:28.278003 12590 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.280529 12631 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.237455 12479 consensus_queue.cc:1059] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260430 07:59:28.301887 9977 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:28.293553 10568 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260430 07:59:28.213258 12619 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:28.307642 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 175822us
W20260430 07:59:28.313256 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 46466.1us
I20260430 07:59:28.314081 9828 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:28.260016 12618 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.288242 12658 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:28.284098 12631 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.269132 12648 consensus_queue.cc:1059] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260430 07:59:28.327464 12670 raft_consensus.cc:3055] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:28.333839 10560 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:28.335286 12664 consensus_queue.cc:1059] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260430 07:59:28.349898 12301 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:28.352427 12648 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:28.353719 12518 raft_consensus.cc:3055] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 8 LEADER]: Stepping down as leader of term 8
W20260430 07:59:28.337843 12532 consensus_meta.cc:287] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.525s user 0.006s sys 0.006s
I20260430 07:59:28.379652 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.380734 9936 raft_consensus.cc:2393] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:28.347086 12670 raft_consensus.cc:740] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:28.364322 9933 raft_consensus.cc:1240] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:28.365178 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:28.376587 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 242262us
I20260430 07:59:28.377852 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:28.361310 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.319970 12478 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:28.392637 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.394249 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.395841 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.396847 9932 raft_consensus.cc:2393] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 9: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:28.398854 9937 raft_consensus.cc:1240] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260430 07:59:28.400184 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.364818 12301 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:28.338433 12256 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.072s
I20260430 07:59:28.401037 12301 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:28.362293 12617 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:28.402271 12617 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.379233 12648 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.408795 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.393597 9934 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:28.361038 12619 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:28.413403 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.321782 12566 consensus_queue.cc:1048] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.064s
I20260430 07:59:28.376034 12518 raft_consensus.cc:740] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:28.417662 12670 consensus_queue.cc:260] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.419152 12670 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:28.414212 9929 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:28.417830 12518 consensus_queue.cc:260] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.435787 12648 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:28.436059 12451 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.437420 12662 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260430 07:59:28.437741 12618 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.443717 9930 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:28.445474 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:28.446007 12657 raft_consensus.cc:3055] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:28.446229 9863 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:28.446404 12657 raft_consensus.cc:740] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:28.446949 12640 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.447871 12657 consensus_queue.cc:260] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.439599 12617 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.476627 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.464216 12518 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:28.477468 9978 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:28.480824 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:28.481581 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:28.482612 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.482704 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:28.483671 9932 raft_consensus.cc:1240] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
W20260430 07:59:28.502720 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 124009us
I20260430 07:59:28.484220 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.485180 12640 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:28.488215 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 101511us
I20260430 07:59:28.488415 12631 raft_consensus.cc:3055] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:28.489449 12369 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:28.497694 12658 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:28.497869 9930 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 8. Current term is 9. Ops: []
I20260430 07:59:28.443993 12453 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.506323 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:28.510061 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:28.507333 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 49651us
I20260430 07:59:28.507473 9828 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:28.517321 12536 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:28.517940 12536 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.518918 9861 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 7: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:28.519680 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:28.520413 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.521787 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:28.498770 12451 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.526155 9856 raft_consensus.cc:2393] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 9: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:28.527101 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:28.530211 9828 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:28.533203 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.536017 9935 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:28.538575 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:28.540782 9933 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 7: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:28.542229 10011 raft_consensus.cc:1275] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:28.514997 12648 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:28.552807 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:28.555752 9930 raft_consensus.cc:2393] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 9: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:28.519646 12657 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:28.545979 9860 raft_consensus.cc:1240] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:28.546252 9862 raft_consensus.cc:1240] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:28.548892 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:28.551482 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:28.504246 12631 raft_consensus.cc:740] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:28.554085 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:28.519829 12623 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 07:59:28.571861 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:28.582041 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.580708 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:28.583194 10004 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:28.573046 9933 raft_consensus.cc:2393] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:28.573770 10010 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:28.557920 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:28.541206 12670 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:28.586068 10007 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:28.545717 12619 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:28.589223 9828 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:28.514911 12369 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.595072 9863 raft_consensus.cc:1275] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:28.523320 12450 consensus_queue.cc:1059] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260430 07:59:28.655356 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:28.576996 12623 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:28.667716 9935 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
W20260430 07:59:28.669615 12674 log.cc:927] Time spent T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
I20260430 07:59:28.671017 12631 consensus_queue.cc:260] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.672605 12631 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:28.523129 12658 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:28.669808 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.716230 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:28.718631 10007 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:28.654032 12640 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:28.575855 12648 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.658174 12549 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:28.703754 12623 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:28.743681 10010 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:28.747898 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:28.752594 9863 raft_consensus.cc:2393] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
W20260430 07:59:28.754999 12675 log.cc:927] Time spent T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.104s user 0.000s sys 0.001s
W20260430 07:59:28.758877 12676 log.cc:927] Time spent T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260430 07:59:28.732650 12670 raft_consensus.cc:697] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:28.737291 12450 raft_consensus.cc:3055] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:28.737836 12654 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260430 07:59:28.776762 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:28.791903 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:28.792830 9935 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
W20260430 07:59:28.794762 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.733865 12640 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.796612 12453 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.775487 12450 raft_consensus.cc:740] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:28.798115 12450 consensus_queue.cc:260] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.799340 12450 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:28.736586 12549 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:28.804497 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.805493 12301 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.806674 12658 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:28.806191 12677 log.cc:927] Time spent T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260430 07:59:28.797169 12640 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.815332 9860 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260430 07:59:28.816622 12256 consensus_queue.cc:1048] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:28.821419 10004 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
W20260430 07:59:28.830849 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.794106 12536 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.808472 12301 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:28.837388 12682 log.cc:927] Time spent T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260430 07:59:28.846532 12648 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:28.825083 12479 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
W20260430 07:59:28.851583 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 80431.5us
W20260430 07:59:28.856118 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 54306us
I20260430 07:59:28.859582 9903 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:28.856837 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 267654us
I20260430 07:59:28.863126 12566 consensus_queue.cc:1048] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:28.883905 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260430 07:59:28.899588 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 80265.5us
I20260430 07:59:28.871126 12619 raft_consensus.cc:697] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:28.917968 12549 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:28.920542 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:28.922803 12679 log.cc:927] Time spent T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.109s user 0.002s sys 0.000s
W20260430 07:59:28.925091 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 58674.9us
I20260430 07:59:28.901140 12369 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:28.899000 12301 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:28.948854 9828 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:28.950752 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:28.951655 12654 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260430 07:59:28.952931 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.961826 9854 raft_consensus.cc:1275] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260430 07:59:28.962458 9854 pending_rounds.cc:85] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 4
I20260430 07:59:28.962766 9854 pending_rounds.cc:107] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260430 07:59:28.963060 9854 raft_consensus.cc:2889] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260430 07:59:28.979297 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:28.983035 9828 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:28.935722 12670 consensus_queue.cc:237] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:28.984319 12623 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:29.005818 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:28.993014 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:29.026604 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 115016us
I20260430 07:59:28.968456 12631 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election won for term 9
W20260430 07:59:29.027586 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 44612.7us
I20260430 07:59:29.034924 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:28.989513 12619 consensus_queue.cc:237] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:29.041872 12518 consensus_meta.cc:287] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.534s user 0.011s sys 0.000s
W20260430 07:59:29.046232 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 186697us
W20260430 07:59:29.049444 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 178119us
I20260430 07:59:29.051756 12631 raft_consensus.cc:697] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:29.052038 9828 leader_election.cc:400] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8.
I20260430 07:59:29.053522 12662 raft_consensus.cc:2749] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:29.054293 12453 consensus_queue.cc:1048] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260430 07:59:29.054728 12662 consensus_queue.cc:1048] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:59:29.055161 12654 consensus_queue.cc:1059] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:59:29.056075 12654 consensus_queue.cc:1059] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:59:29.056348 12453 raft_consensus.cc:3055] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:29.056241 12688 consensus_queue.cc:1059] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:29.056825 12453 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:29.057164 12688 raft_consensus.cc:3055] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Stepping down as leader of term 8
I20260430 07:59:29.057651 12688 raft_consensus.cc:740] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:29.057804 12453 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.058560 12688 consensus_queue.cc:260] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.059149 12453 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:29.059917 12688 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:29.058082 12617 consensus_queue.cc:1059] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:59:29.063354 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.064118 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.075533 9860 raft_consensus.cc:2393] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 9: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
W20260430 07:59:29.080090 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 47519.2us
W20260430 07:59:29.081301 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 47079.8us
I20260430 07:59:29.075825 9856 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:28.969736 12549 consensus_queue.cc:1059] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260430 07:59:28.958995 12301 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:29.093719 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:29.092650 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:29.030331 12566 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:29.099951 9977 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.098845 10003 raft_consensus.cc:1240] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:29.100276 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:29.098740 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.056958 12654 raft_consensus.cc:3055] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:29.056962 12662 consensus_queue.cc:1059] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260430 07:59:29.103961 9904 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:29.105145 10003 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260430 07:59:29.105589 10009 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 7: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:28.991312 12422 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.102s
I20260430 07:59:29.111116 9903 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:29.103914 10007 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260430 07:59:29.107713 9932 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02a63423ae464eb7b415a38419524150"
dest_uuid: "77e19ab653f540339c1c613fad1451b5"
mode: GRACEFUL
new_leader_uuid: "48e81934d52a4e93be5171ceea193032"
from {username='slave'} at 127.0.0.1:52704
I20260430 07:59:29.164064 9828 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.138989 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:29.129122 12479 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:29.153044 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:29.113986 9977 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.180287 9828 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.178356 9932 raft_consensus.cc:606] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Received request to transfer leadership to TS 48e81934d52a4e93be5171ceea193032
I20260430 07:59:29.156080 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:29.052834 12631 consensus_queue.cc:237] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:29.182489 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 71158us
I20260430 07:59:29.181490 9932 raft_consensus.cc:614] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:29.171577 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:29.189879 9977 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.192131 9977 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:59:29.192596 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 92685.5us
I20260430 07:59:29.183001 10004 raft_consensus.cc:2393] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 9: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:29.181509 10012 raft_consensus.cc:2393] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:29.192165 9828 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.198200 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:29.200006 9977 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:29.134660 12476 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:29.107213 12654 raft_consensus.cc:740] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:29.135277 12623 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:29.211352 9828 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.208385 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:29.180136 12689 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:29.184350 10560 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260430 07:59:29.231879 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:29.207859 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 96845.4us
I20260430 07:59:29.233673 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:29.183504 12483 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:29.222658 9859 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:29.112744 12640 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:29.241830 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:29.242969 9930 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 7: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:29.218245 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:29.246690 9938 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:29.191045 12566 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:29.191808 12648 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:29.207144 12518 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:29.216413 12369 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:29.174968 12450 consensus_queue.cc:1048] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20260430 07:59:29.147763 12410 consensus_queue.cc:1048] T 202cb0ccf6a947e6bdcffc383288322c P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.079s
I20260430 07:59:29.216398 12536 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:29.212018 12432 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:29.266718 9929 raft_consensus.cc:1275] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:29.277251 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:29.273346 9938 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:29.223265 12670 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
W20260430 07:59:29.275988 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:29.238760 12619 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election won for term 9
W20260430 07:59:29.285956 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 151297us
I20260430 07:59:29.243793 12623 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:29.286617 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:29.307082 10006 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:29.308413 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:29.309065 10008 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 9.
I20260430 07:59:29.277148 12536 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:29.321630 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 121738us
I20260430 07:59:29.258898 12668 consensus_queue.cc:1059] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260430 07:59:29.292975 12369 raft_consensus.cc:697] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:29.257301 12640 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.236302 12476 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.274788 12566 raft_consensus.cc:697] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:29.275736 12478 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:29.330423 9977 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:59:29.333112 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:29.376880 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:29.276975 12450 consensus_queue.cc:1243] T 829ed212483c445ba0dd4ae328bde3c2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer 48e81934d52a4e93be5171ceea193032 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260430 07:59:29.308187 12670 raft_consensus.cc:697] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:29.323526 9859 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260430 07:59:29.391469 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 61126.9us
I20260430 07:59:29.392168 9977 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:29.331270 12536 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:29.308962 12619 raft_consensus.cc:697] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:29.350957 12369 consensus_queue.cc:237] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:29.290606 12432 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.352334 12618 raft_consensus.cc:3055] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:29.353564 12654 consensus_queue.cc:260] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.320410 12662 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:29.274942 12648 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:29.389371 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 253729us
W20260430 07:59:29.408819 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 205767us
I20260430 07:59:29.413995 10007 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:29.393318 9977 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.388595 12478 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:29.393798 10006 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:29.396564 9903 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:29.420663 9903 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:29.339278 12664 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:29.423288 10007 raft_consensus.cc:606] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:29.424400 10007 raft_consensus.cc:614] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:29.397711 12301 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:29.428609 9977 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.400490 12623 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:29.401157 12705 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:29.428277 12617 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:29.463976 12617 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.466277 12617 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:29.466924 12617 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260430 07:59:29.467986 12707 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:59:29.468765 12707 consensus_queue.cc:1059] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:59:29.474202 12476 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:29.474937 12476 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260430 07:59:29.475495 12476 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:29.405530 12618 raft_consensus.cc:740] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:29.411883 12410 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:29.426626 12478 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:29.447219 12301 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:29.498533 12709 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:29.499480 12566 consensus_queue.cc:237] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.500362 12619 consensus_queue.cc:237] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:29.519479 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 284056us
I20260430 07:59:29.502236 12432 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:29.481323 12689 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:29.428979 12662 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.490892 12670 consensus_queue.cc:237] T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.478586 12705 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:29.482834 12623 raft_consensus.cc:697] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:59:29.589504 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 197566us
I20260430 07:59:29.504043 12479 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:29.505121 12618 consensus_queue.cc:260] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.593952 10012 pending_rounds.cc:85] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 3
I20260430 07:59:29.603785 10012 pending_rounds.cc:107] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260430 07:59:29.604102 10012 raft_consensus.cc:2889] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:29.484151 12708 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:29.441583 12664 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:29.594694 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 166137us
I20260430 07:59:29.596728 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:29.604866 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.490582 12410 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.603293 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.610298 9977 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:29.608942 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 195429us
I20260430 07:59:29.611967 9977 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.610776 9861 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:29.616588 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.617579 9863 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:29.623867 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.625888 9854 raft_consensus.cc:2393] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 8: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:29.600739 12705 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:29.586731 12301 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.631086 9904 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:29.612747 12706 log.cc:927] Time spent T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20260430 07:59:29.632580 9977 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.606268 12692 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:29.648454 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 226311us
I20260430 07:59:29.650383 9903 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:29.651937 9929 raft_consensus.cc:1275] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:29.625602 12479 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:29.636394 12661 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.018s sys 0.043s
I20260430 07:59:29.695367 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:29.596163 12689 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:29.702804 9828 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:29.594237 12709 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:29.714710 12711 log.cc:927] Time spent T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260430 07:59:29.654066 12640 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:29.658124 12618 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:29.719620 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 87089us
I20260430 07:59:29.724414 9828 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:29.724946 9977 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:29.704950 12661 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.637s user 0.018s sys 0.043s
I20260430 07:59:29.732777 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:59:29.749115 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 118091us
I20260430 07:59:29.614830 12708 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.750429 9904 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:29.766340 12709 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:29.767258 12709 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:29.768103 12709 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:29.768724 12709 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:29.780917 10007 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:29.781908 10007 raft_consensus.cc:606] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:29.782414 10007 raft_consensus.cc:614] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:29.698261 12479 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:29.793581 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 143320us
I20260430 07:59:29.797369 10006 raft_consensus.cc:1275] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:29.807730 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.757350 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:29.756557 12710 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:29.760358 12648 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:29.818440 9903 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:29.757351 12733 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
W20260430 07:59:29.824142 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:29.829262 9903 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:29.831054 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:29.796718 12432 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:29.798807 12729 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:29.831117 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.836928 9862 raft_consensus.cc:1240] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:29.731015 12623 consensus_queue.cc:237] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:29.859220 12518 consensus_meta.cc:287] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.597s user 0.010s sys 0.000s
I20260430 07:59:29.848682 12432 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.859928 12716 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:29.849524 12733 raft_consensus.cc:697] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:29.862076 12479 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:29.862661 10006 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:29.892318 10006 raft_consensus.cc:606] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:29.892840 10006 raft_consensus.cc:614] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:29.863816 12740 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260430 07:59:29.865015 12453 consensus_meta.cc:287] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.805s user 0.009s sys 0.003s
I20260430 07:59:29.908370 12740 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:29.822386 12709 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:29.908999 12740 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:29.909574 12714 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:29.878156 12518 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:29.908036 12657 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:29.915231 12657 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.915522 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.917936 12708 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:29.918602 10006 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:29.918815 12708 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260430 07:59:29.919263 10006 raft_consensus.cc:606] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:29.861109 10560 raft_consensus.cc:2393] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:29.920846 12716 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:29.921444 12716 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.923868 12716 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:29.924481 12716 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:29.923909 12657 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:29.872105 12410 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:29.836766 12662 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:29.918973 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.939100 9863 raft_consensus.cc:2410] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260430 07:59:29.919771 10006 raft_consensus.cc:614] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:29.931011 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.889077 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:29.834122 12738 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:29.941154 9854 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:29.886454 12451 raft_consensus.cc:3055] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Stepping down as leader of term 7
W20260430 07:59:29.935823 12536 consensus_meta.cc:287] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.533s user 0.009s sys 0.001s
W20260430 07:59:29.985805 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 48082.6us
W20260430 07:59:30.026432 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 197353us
I20260430 07:59:30.072540 9903 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:29.938342 12301 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:29.925043 12716 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:29.970463 12738 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:30.098416 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1792) took 1063 ms. Trace:
I20260430 07:59:30.029732 12410 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:30.100956 10006 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:30.104528 10006 raft_consensus.cc:606] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:30.030171 12664 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.102710 9857 rpcz_store.cc:276] 0430 07:59:29.034543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:29.034689 (+ 146us) service_pool.cc:225] Handling call
0430 07:59:30.092161 (+1057472us) spinlock_profiling.cc:243] Waited 558 ms on lock 0x7b54001cd9d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:30.098323 (+ 6162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1282865024}
I20260430 07:59:30.105505 9903 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:29.996770 12451 raft_consensus.cc:740] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:30.113168 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 388217us
I20260430 07:59:29.869133 12729 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.110626 9978 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:30.105455 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 490160us
I20260430 07:59:30.111697 9903 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:30.125818 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 393059us
I20260430 07:59:30.125478 9903 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:30.114727 9977 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:30.131276 10003 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 9. Current term is 10. Ops: []
I20260430 07:59:30.133634 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.134480 10003 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:30.135098 10010 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: []
W20260430 07:59:30.189649 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 74979.9us
I20260430 07:59:30.190858 9977 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:30.192430 9977 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:30.104568 12738 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:30.108632 12658 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:59:30.122952 12708 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:30.268630 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.081545 12536 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.196036 12689 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:30.331048 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:30.294153 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 169114us
W20260430 07:59:30.288363 12661 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.026s sys 0.051s
I20260430 07:59:30.341586 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.342967 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.345242 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.346900 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.361533 10007 raft_consensus.cc:2393] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 7: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:30.371538 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.383885 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.342772 12714 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:30.345423 12661 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s user 0.026s sys 0.052s
I20260430 07:59:30.406682 12728 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260430 07:59:30.417729 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.422008 12737 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:30.422433 12745 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:30.422786 12750 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:30.355463 12657 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:30.114355 12733 consensus_queue.cc:237] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.429400 12714 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.437750 12747 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:30.442524 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.331477 12708 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.279187 12451 consensus_queue.cc:260] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.465831 10007 raft_consensus.cc:2410] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260430 07:59:30.418149 12478 consensus_meta.cc:287] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.904s user 0.014s sys 0.001s
I20260430 07:59:30.406916 12432 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.439970 12745 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:30.469643 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 273082us
I20260430 07:59:30.470317 10009 raft_consensus.cc:2393] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 7: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:30.458859 12739 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:30.454692 12728 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:30.462023 12716 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.488605 12451 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:30.459297 12657 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.479444 12745 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
W20260430 07:59:30.510463 12715 log.cc:927] Time spent T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.438s user 0.005s sys 0.000s
I20260430 07:59:30.495558 12739 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:30.532981 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:30.542009 9937 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:30.533237 9930 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260430 07:59:30.542727 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:30.533146 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:30.499785 12689 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.536130 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:30.537722 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:30.539525 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:30.510366 12727 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:30.510645 12743 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:30.509518 12759 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:30.552325 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:30.558832 12759 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.560015 9862 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:30.561019 12432 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.561311 12764 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.553030 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:30.548029 9933 raft_consensus.cc:1275] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260430 07:59:30.562983 12760 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.561833 12432 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:30.563911 12760 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:30.564580 12660 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.510566 12723 raft_consensus.cc:2749] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:30.567414 9903 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:30.476130 12478 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.585037 10003 raft_consensus.cc:2393] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:30.456738 12750 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.583343 9904 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:30.544481 12739 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:30.468885 12747 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.548316 12566 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:30.586006 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 371640us
I20260430 07:59:30.556595 12658 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:30.570685 12536 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.572027 12729 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.565227 12759 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.637104 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.679075 10009 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:30.588680 12737 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.677865 10011 raft_consensus.cc:1240] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:30.509228 12728 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.680601 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:30.626257 12726 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:30.568511 12716 consensus_queue.cc:1048] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260430 07:59:30.589529 12718 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election won for term 6
W20260430 07:59:30.695642 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 64048.8us
I20260430 07:59:30.557163 12689 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:30.686962 9903 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:30.688473 9935 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
W20260430 07:59:30.694259 12479 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s user 0.020s sys 0.021s
W20260430 07:59:30.634161 12670 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.766s user 0.008s sys 0.007s
W20260430 07:59:30.690917 12735 log.cc:927] Time spent T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.527s user 0.000s sys 0.002s
I20260430 07:59:30.562032 12764 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:30.722702 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.684680 12765 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.690572 12729 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:30.720878 9977 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:30.689227 12723 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.701053 12536 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.733781 12752 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:30.567232 12727 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:30.734135 12772 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.639968 12566 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:30.734452 12734 log.cc:927] Time spent T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.265s user 0.000s sys 0.002s
W20260430 07:59:30.720345 12479 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s user 0.020s sys 0.021s
W20260430 07:59:30.750136 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 169423us
I20260430 07:59:30.614503 12660 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:30.732465 12670 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.861s user 0.008s sys 0.008s
I20260430 07:59:30.762696 10568 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260430 07:59:30.699723 12658 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.766273 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:30.746388 12708 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.740991 12754 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.743011 12718 raft_consensus.cc:697] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:30.698199 12737 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:30.739013 12536 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:30.772552 12740 consensus_meta.cc:287] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.863s user 0.014s sys 0.000s
I20260430 07:59:30.759536 12714 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.739311 12729 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:30.812175 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 60502.8us
I20260430 07:59:30.809031 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1794) took 1393 ms. Trace:
W20260430 07:59:30.890848 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:30.775589 12657 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:30.764200 12751 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.804525 12479 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:30.916510 9904 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:30.917037 9859 rpcz_store.cc:276] 0430 07:59:29.415232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:29.415629 (+ 397us) service_pool.cc:225] Handling call
0430 07:59:29.416043 (+ 414us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:29.641634 (+225591us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:29.641681 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:29.642276 (+ 595us) log.cc:844] Serialized 26 byte log entry
0430 07:59:29.722039 (+ 79763us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:29.722076 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:29.722155 (+ 79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:30.808817 (+1086662us) raft_consensus.cc:1685] finished
0430 07:59:30.808836 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:30.808975 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":48135,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"thread_start_us":304489,"threads_started":2,"wal-append.queue_time_us":697229}
W20260430 07:59:30.940531 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:30.940580 9904 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:30.944868 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 175675us
I20260430 07:59:30.815804 12658 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.916181 12716 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.929638 12714 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:30.946870 12728 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.946522 12478 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:30.965068 12708 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:30.969019 12714 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.969606 12708 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.953950 12760 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:30.961802 12766 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:30.974038 12728 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:30.982637 12773 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:30.913622 12740 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.965595 12753 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:30.981117 12747 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:30.985262 12756 log.cc:927] Time spent T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.291s user 0.000s sys 0.001s
I20260430 07:59:30.973474 12432 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:31.001099 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 252512us
I20260430 07:59:30.973665 12763 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:30.811859 12743 consensus_queue.cc:1048] T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.192s
I20260430 07:59:30.991963 12776 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:30.966331 12716 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:30.927870 12726 consensus_queue.cc:1059] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.077s
I20260430 07:59:30.999032 12689 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.018074 12780 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:30.998174 12750 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.019158 12767 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.038647 12791 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:30.809911 12754 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:30.983248 12773 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:30.742825 12752 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:31.048923 12660 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:30.742192 12765 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:30.749300 12772 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:31.049521 12777 log.cc:927] Time spent T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.230s user 0.000s sys 0.001s
W20260430 07:59:31.050527 12771 log.cc:927] Time spent T d2df038061ff495dbdb2eaa4ad73c76e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20260430 07:59:30.813628 12718 consensus_queue.cc:237] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:31.051903 12746 log.cc:927] Time spent T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.283s user 0.000s sys 0.001s
I20260430 07:59:30.741828 12723 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:30.947113 12751 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:31.016088 12776 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:31.019712 12767 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.013213 12566 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:30.993944 12766 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260430 07:59:31.016181 12764 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.021945 12478 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:31.025409 10532 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.008s sys 0.070s
W20260430 07:59:31.059405 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 118874us
I20260430 07:59:31.003238 12753 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:31.045661 12657 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:59:31.068400 12774 log.cc:927] Time spent T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20260430 07:59:31.041764 12791 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.074079 12740 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.075816 12754 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.078727 12708 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.125099 12707 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:31.083202 10011 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1182) took 1494 ms. Trace:
I20260430 07:59:31.076951 12479 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:31.125718 12719 log.cc:927] Time spent T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.687s user 0.003s sys 0.000s
I20260430 07:59:31.125756 12566 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:31.127207 12566 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.086120 12765 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.086495 12758 raft_consensus.cc:3055] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:31.037032 12689 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:31.129750 12758 raft_consensus.cc:740] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:31.130141 12689 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:31.131939 12618 consensus_meta.cc:287] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 1.404s user 0.016s sys 0.000s
I20260430 07:59:31.126137 10011 rpcz_store.cc:276] 0430 07:59:29.589119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:29.589307 (+ 188us) service_pool.cc:225] Handling call
0430 07:59:29.589645 (+ 338us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:30.690385 (+1100740us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:31.082917 (+392532us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400147654. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:31.083135 (+ 218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2422683520}
I20260430 07:59:31.133777 12791 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:31.134362 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.135700 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.136649 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1183) took 1547 ms. Trace:
I20260430 07:59:31.130980 12758 consensus_queue.cc:260] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.138680 12758 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:31.110824 9904 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:31.137199 10012 rpcz_store.cc:276] 0430 07:59:29.589614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:29.589796 (+ 182us) service_pool.cc:225] Handling call
0430 07:59:29.590263 (+ 467us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:29.700031 (+109768us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:29.700084 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:29.733732 (+ 33648us) log.cc:844] Serialized 52 byte log entry
0430 07:59:29.933753 (+200021us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:29.933811 (+ 58us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:29.933908 (+ 97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:31.136314 (+1202406us) raft_consensus.cc:1685] finished
0430 07:59:31.136350 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:31.136562 (+ 212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52,"raft.queue_time_us":156559,"raft.run_cpu_time_us":64,"raft.run_wall_time_us":68,"thread_start_us":221374,"threads_started":2,"wal-append.queue_time_us":852748}
W20260430 07:59:31.090365 12738 consensus_meta.cc:287] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.798s user 0.005s sys 0.008s
I20260430 07:59:31.139818 12738 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.067168 12536 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.125806 12707 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:31.141229 12768 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:31.142127 12768 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:31.144461 12738 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.145385 12566 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.143314 12707 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.136893 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.146824 10011 raft_consensus.cc:2393] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:31.147229 12657 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.148699 12723 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.149499 12657 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:31.112660 10532 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.811s user 0.009s sys 0.072s
I20260430 07:59:31.150815 12714 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:31.119226 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 146866us
I20260430 07:59:31.155172 12767 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:31.146644 12752 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:31.095443 12776 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:31.101886 12729 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.089128 12737 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.163527 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:31.147846 12745 consensus_meta.cc:287] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.597s user 0.008s sys 0.000s
I20260430 07:59:31.168215 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.171254 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:31.118909 12727 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:31.177943 9854 raft_consensus.cc:1275] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260430 07:59:31.170512 10012 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
W20260430 07:59:31.179951 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 69352us
W20260430 07:59:31.182188 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 230637us
I20260430 07:59:31.190240 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.191054 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:31.191855 10012 raft_consensus.cc:2393] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 7: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:31.188167 12536 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:31.196637 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.196976 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:31.198240 12451 consensus_meta.cc:287] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.656s user 0.012s sys 0.000s
W20260430 07:59:31.199800 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:31.146828 12751 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.204272 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.205255 9854 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:31.205929 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.207060 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.211794 10007 raft_consensus.cc:2393] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 7: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:31.213721 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.216439 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.217669 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.148727 12689 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.117149 12478 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:31.222894 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.223625 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.218536 9857 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:31.229084 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.228724 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:31.230652 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.231135 9933 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:31.231230 9937 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:31.231341 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.232721 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:31.233148 10007 raft_consensus.cc:2410] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260430 07:59:31.233566 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.230124 9936 raft_consensus.cc:2410] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260430 07:59:31.234413 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.231590 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.237407 9930 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:31.239701 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.244186 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.219430 12758 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.247244 9933 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 10.
I20260430 07:59:31.223548 12451 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.243360 10007 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:31.262115 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.262812 9934 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:31.264223 12773 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.270961 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.271371 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.276510 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:31.284258 10007 raft_consensus.cc:2393] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 7: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:31.284698 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.285190 9854 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:31.178381 12745 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.274039 12451 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.289707 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.293434 10561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.295146 10562 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:31.306383 12705 consensus_meta.cc:287] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 1.647s user 0.015s sys 0.000s
I20260430 07:59:31.295965 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.307474 9935 raft_consensus.cc:1275] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260430 07:59:31.282357 12740 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:31.432601 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 50318.8us
I20260430 07:59:31.303300 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:31.300496 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.386523 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.298586 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.440603 9978 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:31.442787 12751 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.387912 12766 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.297545 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.443252 12705 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.298656 12784 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.154s
I20260430 07:59:31.462687 9904 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:59:31.416822 12736 log.cc:927] Time spent T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.805s user 0.000s sys 0.005s
I20260430 07:59:31.297654 10534 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:31.450730 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:31.471750 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 174138us
I20260430 07:59:31.473057 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.472401 12590 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:31.475557 12590 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:31.475775 12787 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:31.475966 12590 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:31.476356 12787 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:31.476778 12787 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:31.462240 12740 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:31.482287 12730 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.447656 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.448246 12722 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.450191 12751 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.483960 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1137) took 2277 ms. Trace:
I20260430 07:59:31.484193 12722 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.484411 9932 rpcz_store.cc:276] 0430 07:59:29.206350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:29.209390 (+ 3040us) service_pool.cc:225] Handling call
0430 07:59:29.209998 (+ 608us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:29.645918 (+435920us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:29.645968 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:29.646784 (+ 816us) log.cc:844] Serialized 52 byte log entry
0430 07:59:29.758569 (+111785us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:29.795632 (+ 37063us) log.cc:844] Serialized 14 byte log entry
0430 07:59:29.828071 (+ 32439us) log.cc:844] Serialized 14 byte log entry
0430 07:59:30.003140 (+175069us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:30.003255 (+ 115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:31.408421 (+1405166us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:31.483632 (+ 75211us) raft_consensus.cc:1685] finished
0430 07:59:31.483661 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:31.483807 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65828,"prepare.queue_time_us":63114,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":91,"raft.queue_time_us":252077,"raft.run_cpu_time_us":413,"raft.run_wall_time_us":61862,"spinlock_wait_cycles":8238976,"thread_start_us":711696,"threads_started":3,"wal-append.queue_time_us":869709}
I20260430 07:59:31.383399 12689 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.487066 12689 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.489655 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.434660 12742 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.491405 12751 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.491694 12742 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.494174 12742 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:31.304697 12478 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.456723 12768 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.458163 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.420018 12772 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.495884 9854 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:31.267869 12758 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.496526 12740 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.496843 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.497484 9854 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:31.498294 12758 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.498440 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:31.307189 12739 consensus_meta.cc:287] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.668s user 0.011s sys 0.000s
I20260430 07:59:31.455947 12743 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:31.499137 12739 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.499609 12743 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:31.500063 12743 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:31.500198 12689 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.290401 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.307132 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.460532 12776 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.434341 12716 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:31.452502 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:31.303648 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.303344 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.362205 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:31.466535 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:31.468804 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.483124 12730 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.489809 12722 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:31.499294 9854 raft_consensus.cc:2410] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260430 07:59:31.460131 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.533833 9858 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:31.303095 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.510708 10010 raft_consensus.cc:2368] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 8. Current term is 9.
I20260430 07:59:31.510751 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1140) took 1895 ms. Trace:
I20260430 07:59:31.535501 9938 rpcz_store.cc:276] 0430 07:59:29.615793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:29.615982 (+ 189us) service_pool.cc:225] Handling call
0430 07:59:29.616437 (+ 455us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:29.799234 (+182797us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:29.799280 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:29.799336 (+ 56us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:29.799385 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:31.510532 (+1711147us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b5400337cd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:31.510576 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:31.510696 (+ 120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":419769216}
I20260430 07:59:31.536859 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.537457 9978 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:31.537762 9938 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:31.539215 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.539705 9930 raft_consensus.cc:2468] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:31.513686 10004 raft_consensus.cc:1240] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:31.517009 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.523864 9863 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:31.503769 9937 raft_consensus.cc:2393] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:31.528144 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:31.529433 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:31.544992 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:31.544847 10007 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:31.547626 9932 raft_consensus.cc:2393] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:31.543138 9903 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:31.522897 9856 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:31.511059 9977 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:31.451810 12766 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:31.551075 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:32.348055 10012 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:31.641163 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:31.584398 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:31.579877 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:31.644610 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.350363 9930 raft_consensus.cc:2410] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260430 07:59:32.016129 9859 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:32.351809 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1806) took 1049 ms. Trace:
W20260430 07:59:32.347224 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 803831us
I20260430 07:59:32.352522 9859 rpcz_store.cc:276] 0430 07:59:31.302321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.452206 (+149885us) service_pool.cc:225] Handling call
0430 07:59:32.351725 (+899519us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:31.583395 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:32.354673 9903 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:31.749761 12689 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:32.346244 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 835066us
I20260430 07:59:32.356176 12689 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.356835 9977 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:32.347941 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:32.347335 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:32.117790 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:32.350740 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:32.350484 9863 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:32.359068 12689 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.558713 9860 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 2.
I20260430 07:59:32.347606 10533 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:32.362432 9935 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:32.441642 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1802) took 1141 ms. Trace:
I20260430 07:59:32.442239 9860 rpcz_store.cc:276] 0430 07:59:31.300370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.302797 (+ 2427us) service_pool.cc:225] Handling call
0430 07:59:32.441572 (+1138775us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:32.447233 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1810) took 3117 ms. Trace:
I20260430 07:59:31.505491 12776 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.447741 10008 rpcz_store.cc:276] 0430 07:59:29.330112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:29.330268 (+ 156us) service_pool.cc:225] Handling call
0430 07:59:29.334594 (+ 4326us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:29.642853 (+308259us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:29.642904 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:29.643810 (+ 906us) log.cc:844] Serialized 26 byte log entry
0430 07:59:29.735337 (+ 91527us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:29.735413 (+ 76us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:32.347103 (+2611690us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54003c1754. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:32.347159 (+ 56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:32.347213 (+ 54us) raft_consensus.cc:1685] finished
0430 07:59:32.347239 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:32.447152 (+ 99913us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9065,"raft.queue_time_us":35347,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"spinlock_wait_cycles":289633536,"thread_start_us":262905,"threads_started":2,"wal-append.queue_time_us":192932,"wal-append.run_cpu_time_us":4823,"wal-append.run_wall_time_us":2073311}
I20260430 07:59:32.483500 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1817) took 1006 ms. Trace:
I20260430 07:59:31.501237 12739 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:32.296303 12743 consensus_meta.cc:287] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.796s user 0.013s sys 0.000s
I20260430 07:59:32.484073 9857 rpcz_store.cc:276] 0430 07:59:31.476857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.609115 (+132258us) service_pool.cc:225] Handling call
0430 07:59:32.482741 (+873626us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:32.483011 (+ 270us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:32.483074 (+ 63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:32.483141 (+ 67us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:32.483194 (+ 53us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:32.483275 (+ 81us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:32.483427 (+ 152us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:32.489004 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 141479us
I20260430 07:59:31.510212 12451 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:31.609557 12722 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:31.583395 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:31.514642 12705 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:31.548559 12648 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:31.551290 12716 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.548682 12476 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:32.135114 12767 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.870s user 0.023s sys 0.062s
I20260430 07:59:31.622851 12718 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.945094 12730 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.118088 12772 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:31.611841 12740 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:31.583173 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:32.446451 12479 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:32.298810 12745 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:31.548131 12787 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:31.968087 12590 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:32.499898 12618 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s user 0.011s sys 0.011s
I20260430 07:59:32.502986 12743 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.516923 12799 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:32.514555 12767 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.272s user 0.024s sys 0.064s
I20260430 07:59:32.505726 12692 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:32.505741 12648 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:32.521199 10007 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 9.
I20260430 07:59:32.519579 12476 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.519794 12705 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:32.529517 12730 raft_consensus.cc:2749] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:32.522609 12722 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:32.539785 12618 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.807s user 0.012s sys 0.012s
I20260430 07:59:32.516196 12716 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.539997 12740 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:32.567255 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1850) took 1038 ms. Trace:
I20260430 07:59:32.548363 12788 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:32.551218 12648 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:32.534993 12479 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.523322 12718 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.528563 12772 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.568053 10007 rpcz_store.cc:276] 0430 07:59:31.529258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.584573 (+ 55315us) service_pool.cc:225] Handling call
0430 07:59:32.566975 (+982402us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:32.588994 12685 raft_consensus.cc:3055] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:32.589680 12685 raft_consensus.cc:740] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:32.589772 12713 log.cc:927] Time spent T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260430 07:59:32.590567 12685 consensus_queue.cc:260] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.591699 12685 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:32.591830 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1139) took 3376 ms. Trace:
W20260430 07:59:32.593071 12775 log.cc:927] Time spent T 058d2a87aec64ba1b745c5bd593c00e2 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 1.859s user 0.020s sys 0.020s
I20260430 07:59:32.592306 9931 rpcz_store.cc:276] 0430 07:59:29.215075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:29.215283 (+ 208us) service_pool.cc:225] Handling call
0430 07:59:29.236988 (+ 21705us) raft_consensus.cc:1471] Updating replica for 3 ops
0430 07:59:29.237475 (+ 487us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:29.237521 (+ 46us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0430 07:59:29.260769 (+ 23248us) log.cc:844] Serialized 78 byte log entry
0430 07:59:29.482209 (+221440us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:29.518110 (+ 35901us) log.cc:844] Serialized 14 byte log entry
0430 07:59:29.591006 (+ 72896us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:29.591115 (+ 109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:31.078856 (+1487741us) spinlock_profiling.cc:243] Waited 6.76 ms on lock 0x7b1800352540. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661e5b62 00007fb5661e7e01 00007fb5661e7db9 00007fb5661e7d49 00007fb5661e7d11 00007fb5661e700d 0000000000379254 0000000000379099 00007fb5661c8fba
0430 07:59:32.489719 (+1410863us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b1800352540. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661e5a33 00007fb5661e7e01 00007fb5661e7db9 00007fb5661e7d49 00007fb5661e7d11 00007fb5661e700d 0000000000379254 0000000000379099 00007fb5661c8fba
0430 07:59:32.591598 (+101879us) raft_consensus.cc:1685] finished
0430 07:59:32.591621 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:32.591769 (+ 148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":35525,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":2018,"raft.queue_time_us":119195,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":12848,"spinlock_wait_cycles":19930368,"thread_start_us":148326,"threads_started":1,"wal-append.queue_time_us":281196}
I20260430 07:59:32.551636 12692 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.598532 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.591154 12476 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.600889 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.603037 9863 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:32.605158 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.606294 9861 raft_consensus.cc:2410] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260430 07:59:32.571877 12705 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.572207 12722 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.582849 12740 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.617764 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:32.618477 9861 raft_consensus.cc:2393] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 9: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:32.613700 12766 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.634923 12590 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.635514 12716 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.637451 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:32.636314 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:32.643716 10565 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54770 (request call id 395) took 1353 ms. Trace:
I20260430 07:59:32.644340 12722 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.645123 10565 rpcz_store.cc:276] 0430 07:59:31.290681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.299499 (+ 8818us) service_pool.cc:225] Handling call
0430 07:59:32.566188 (+1266689us) spinlock_profiling.cc:243] Waited 7.89 ms on lock 0x7b58001f1798. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:32.643643 (+ 77455us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18148736}
I20260430 07:59:32.665495 10012 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:32.636817 12787 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.639748 12743 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.675510 12640 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:32.676733 12705 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:32.677721 12807 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20260430 07:59:32.682046 12718 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.683596 12740 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:32.667172 12806 log.cc:927] Time spent T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
I20260430 07:59:32.685878 10566 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54770 (request call id 402) took 1170 ms. Trace:
I20260430 07:59:32.691190 10012 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1851) took 1161 ms. Trace:
I20260430 07:59:32.702858 10012 rpcz_store.cc:276] 0430 07:59:31.529545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.550781 (+ 21236us) service_pool.cc:225] Handling call
0430 07:59:32.691099 (+1140318us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:32.693521 12479 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:32.695158 12808 log.cc:927] Time spent T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
W20260430 07:59:32.668464 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 1.02834e+06us
I20260430 07:59:32.675657 12772 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.702407 10566 rpcz_store.cc:276] 0430 07:59:31.515622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.543198 (+ 27576us) service_pool.cc:225] Handling call
0430 07:59:31.668837 (+125639us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:31.832589 (+163752us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:31.832637 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:31.891417 (+ 58780us) log.cc:844] Serialized 26 byte log entry
0430 07:59:32.547644 (+656227us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:32.547692 (+ 48us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:32.547773 (+ 81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:32.685625 (+137852us) raft_consensus.cc:1685] finished
0430 07:59:32.685646 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:32.685790 (+ 144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88497,"raft.queue_time_us":169412,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"thread_start_us":430405,"threads_started":1,"wal-append.queue_time_us":688604}
I20260430 07:59:32.714929 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:32.715190 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 219849us
I20260430 07:59:32.699712 12688 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:32.718400 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.719637 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.721580 12816 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:32.722493 12816 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:32.701366 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1816) took 1224 ms. Trace:
I20260430 07:59:32.726662 12718 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:32.732043 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:32.732396 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.733574 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:32.734398 9857 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:32.709826 12813 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:32.737354 9854 raft_consensus.cc:2410] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:32.737685 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1572) took 1507 ms. Trace:
W20260430 07:59:32.745872 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 391192us
I20260430 07:59:32.714673 12479 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:32.713037 12692 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.699353 12640 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.748145 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:32.749308 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 392486us
I20260430 07:59:32.746460 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.728193 12718 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.745862 9937 rpcz_store.cc:276] 0430 07:59:31.230622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.233365 (+ 2743us) service_pool.cc:225] Handling call
0430 07:59:32.737504 (+1504139us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54002f1054. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:32.737629 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":618027392}
I20260430 07:59:32.759411 9856 raft_consensus.cc:2410] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260430 07:59:32.730083 12743 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:32.765829 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.753594 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 384) took 1363 ms. Trace:
I20260430 07:59:32.754068 9978 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:32.761801 9903 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:32.760038 9977 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:32.767647 9936 rpcz_store.cc:276] 0430 07:59:31.390488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.392707 (+ 2219us) service_pool.cc:225] Handling call
0430 07:59:32.752827 (+1360120us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:59:32.753056 (+ 229us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:32.753288 (+ 232us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:32.753323 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:32.753362 (+ 39us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:32.753389 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:32.753441 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:32.753540 (+ 99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4646784}
I20260430 07:59:32.768901 12815 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 07:59:32.769423 12743 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.769464 12815 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:32.753755 12709 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:32.770927 12815 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:32.771206 12709 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:32.771672 12743 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.737075 12688 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:32.773986 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:32.775106 12590 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:32.775629 12590 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.752727 12822 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:32.783850 9903 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:32.785588 12743 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:32.757784 12483 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:32.777413 12617 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:32.789786 12617 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.790884 12640 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:32.792659 10003 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1817) took 2452 ms (client timeout 1829 ms). Trace:
W20260430 07:59:32.793140 10003 rpcz_store.cc:269] 0430 07:59:30.340495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:30.346603 (+ 6108us) service_pool.cc:225] Handling call
0430 07:59:32.792413 (+2445810us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54002df5d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:32.792572 (+ 159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":518043392}
I20260430 07:59:32.788839 12483 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.800648 12483 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.801323 12718 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.802592 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.803529 9861 raft_consensus.cc:2410] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260430 07:59:32.803898 12654 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:32.804505 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.803442 10006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:55676 (request call id 35) took 2702 ms. Trace:
I20260430 07:59:32.811023 10005 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1818) took 2470 ms. Trace:
I20260430 07:59:32.814705 10005 rpcz_store.cc:276] 0430 07:59:30.340770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:30.372849 (+ 32079us) service_pool.cc:225] Handling call
0430 07:59:30.407435 (+ 34586us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:30.459552 (+ 52117us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:30.459585 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:30.459662 (+ 77us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:30.459694 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:32.810691 (+2350997us) spinlock_profiling.cc:243] Waited 51.8 ms on lock 0x7b5400048dd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:32.810757 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:32.810939 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":119242112}
I20260430 07:59:32.816421 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 385) took 1425 ms. Trace:
I20260430 07:59:32.805241 9861 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:32.818202 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.818979 9861 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:32.811836 12590 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.805445 10006 rpcz_store.cc:276] 0430 07:59:30.100626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:30.100805 (+ 179us) service_pool.cc:225] Handling call
0430 07:59:32.803066 (+2702261us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b54000dbea0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569200900 00007fb56927da32 00007fb56e259cca 00007fb5677d2934 00007fb5677d28c2 00007fb5677d27f1 00007fb5677d276c 00007fb5677d1a22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:32.803363 (+ 297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2253470464}
I20260430 07:59:32.749626 12813 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.826038 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260430 07:59:32.757798 12479 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.728760 9855 rpcz_store.cc:276] 0430 07:59:31.476560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.543108 (+ 66548us) service_pool.cc:225] Handling call
0430 07:59:32.360136 (+817028us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:32.360395 (+ 259us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:32.514082 (+153687us) log.cc:844] Serialized 14 byte log entry
0430 07:59:32.700998 (+186916us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:32.701052 (+ 54us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:32.701097 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:32.701158 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:32.701308 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63,"prepare.queue_time_us":242636,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":1198,"raft.queue_time_us":222,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":7463,"spinlock_wait_cycles":5321088,"thread_start_us":339580,"threads_started":2,"wal-append.queue_time_us":198518}
W20260430 07:59:32.813499 10010 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1815) took 2473 ms (client timeout 1928 ms). Trace:
W20260430 07:59:32.827621 10010 rpcz_store.cc:269] 0430 07:59:30.339871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:30.341063 (+ 1192us) service_pool.cc:225] Handling call
0430 07:59:32.813202 (+2472139us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400147154. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:32.813407 (+ 205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2681146624}
W20260430 07:59:32.827975 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 68094.4us
I20260430 07:59:32.829396 9977 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:32.788977 12753 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:32.816982 9933 rpcz_store.cc:276] 0430 07:59:31.390772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.391237 (+ 465us) service_pool.cc:225] Handling call
0430 07:59:32.815443 (+1424206us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8a3e 000000000035a3c9 00007fb56e1e08e0 00007fb56e2a9784 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:59:32.815675 (+ 232us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:32.815948 (+ 273us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:32.815985 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:32.816073 (+ 88us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:32.816117 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:32.816196 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:32.816342 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4356736}
I20260430 07:59:32.838737 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:32.841671 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 57921.2us
I20260430 07:59:32.843055 9903 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:32.847489 9904 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:32.852391 9857 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:32.850322 9977 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:32.825095 12813 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.778043 12716 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:32.836845 12739 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:32.866175 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.862498 9977 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:59:32.873498 12476 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:32.894502 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.874055 12739 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:32.874307 12716 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.777688 12822 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:32.901970 9904 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:32.902547 9854 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:32.903534 12816 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:32.903360 12815 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.904580 12614 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:32.904836 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1582) took 1397 ms. Trace:
I20260430 07:59:32.949946 9932 rpcz_store.cc:276] 0430 07:59:31.507443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.549861 (+ 42418us) service_pool.cc:225] Handling call
0430 07:59:31.550236 (+ 375us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:32.446782 (+896546us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:32.446826 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:32.487692 (+ 40866us) log.cc:844] Serialized 26 byte log entry
0430 07:59:32.761465 (+273773us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:32.770648 (+ 9183us) log.cc:844] Serialized 14 byte log entry
0430 07:59:32.830447 (+ 59799us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:32.830578 (+ 131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:32.904534 (+ 73956us) raft_consensus.cc:1685] finished
0430 07:59:32.904557 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:32.904753 (+ 196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":169570,"prepare.queue_time_us":8703,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":883953,"raft.run_cpu_time_us":407,"raft.run_wall_time_us":123643,"thread_start_us":248805,"threads_started":1,"wal-append.queue_time_us":290146}
I20260430 07:59:32.872340 12718 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Leader election won for term 10
I20260430 07:59:32.905928 12617 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.906983 12453 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:32.873693 12784 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:32.874181 12766 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:32.910547 12648 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:32.971716 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1142) took 2446 ms. Trace:
I20260430 07:59:32.926949 9856 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:32.972198 9929 rpcz_store.cc:276] 0430 07:59:30.525485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:30.525842 (+ 357us) service_pool.cc:225] Handling call
0430 07:59:30.554842 (+ 29000us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:30.685435 (+130593us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:30.685480 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:30.685539 (+ 59us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:30.685572 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:32.971469 (+2285897us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b5400323cd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:32.971515 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:32.971632 (+ 117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":299480832}
W20260430 07:59:32.933218 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 70840.8us
W20260430 07:59:32.938616 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:32.938741 9862 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:32.939370 9903 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:32.940462 9978 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:32.938944 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:32.976227 9859 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 7
I20260430 07:59:32.852674 12709 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:32.943358 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:32.989384 12483 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260430 07:59:32.943845 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 98950us
I20260430 07:59:32.856986 12707 consensus_queue.cc:1059] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20260430 07:59:32.966231 12453 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:32.991839 9858 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1804) took 1690 ms (client timeout 1909 ms). Trace:
W20260430 07:59:32.992281 9858 rpcz_store.cc:269] 0430 07:59:31.301499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.303016 (+ 1517us) service_pool.cc:225] Handling call
0430 07:59:32.991668 (+1688652us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b540029cf54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:32.991770 (+ 102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111077760}
W20260430 07:59:32.993455 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 113282us
I20260430 07:59:32.993590 12709 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:32.994354 12709 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:32.994870 9978 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:32.995888 9978 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:32.996284 12737 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election won for term 9
I20260430 07:59:32.996595 12256 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:32.996898 12737 raft_consensus.cc:697] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 9 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:32.997198 12256 raft_consensus.cc:697] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 8 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:32.997524 12422 raft_consensus.cc:3055] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:32.939733 12722 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:32.997804 12617 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:32.997969 12422 raft_consensus.cc:740] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:32.998179 12722 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:32.998651 12722 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:32.999011 12422 consensus_queue.cc:260] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.000360 12422 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:32.956789 12815 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.941654 12739 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:32.998190 12256 consensus_queue.cc:237] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:32.940223 12758 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:33.002604 10004 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1188) took 2871 ms. Trace:
I20260430 07:59:32.961464 12479 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:32.968317 12718 raft_consensus.cc:697] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 10 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:33.003005 10004 rpcz_store.cc:276] 0430 07:59:30.131404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:30.131591 (+ 187us) service_pool.cc:225] Handling call
0430 07:59:30.131939 (+ 348us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:31.542297 (+1410358us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:33.002345 (+1460048us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54002c5454. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:33.002540 (+ 195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3176652032}
I20260430 07:59:33.004097 12718 consensus_queue.cc:237] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.007592 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.008080 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:33.008797 10011 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1848) took 1480 ms (client timeout 1576 ms). Trace:
I20260430 07:59:32.977034 12766 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.010774 12766 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.987138 12716 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.994879 12709 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:33.012423 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:32.957259 12614 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:32.959309 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 77462.4us
W20260430 07:59:33.011617 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:33.016398 9904 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:59:33.009178 10011 rpcz_store.cc:269] 0430 07:59:31.528504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.528615 (+ 111us) service_pool.cc:225] Handling call
0430 07:59:33.007922 (+1479307us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b58000a2098. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:33.008744 (+ 822us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9338624}
I20260430 07:59:32.940200 12640 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:32.992962 12453 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:33.013650 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 381) took 1624 ms. Trace:
W20260430 07:59:33.023036 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 47661.2us
I20260430 07:59:33.023811 9977 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:33.023314 9934 rpcz_store.cc:276] 0430 07:59:31.389441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:31.389631 (+ 190us) service_pool.cc:225] Handling call
0430 07:59:33.012847 (+1623216us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b5800002d98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:59:33.013019 (+ 172us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:33.013228 (+ 209us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:33.013252 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:33.013395 (+ 143us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:33.013432 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:33.013498 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:33.013598 (+ 100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4387200}
I20260430 07:59:32.992972 9859 raft_consensus.cc:2468] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 7.
I20260430 07:59:33.018451 9904 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:33.086033 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 140764us
I20260430 07:59:33.081179 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:33.086459 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 62708.8us
I20260430 07:59:33.087406 10534 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:33.025840 9828 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:33.088469 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 62737.2us
W20260430 07:59:32.977785 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 96981.5us
I20260430 07:59:33.021262 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.089972 9903 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.090284 9856 raft_consensus.cc:2393] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 7: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:33.082330 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.018862 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.091313 9855 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:33.044749 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.025646 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:33.010946 12822 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:32.968684 12799 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 07:59:33.094973 9857 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 7.
I20260430 07:59:33.105846 10534 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:33.095062 9860 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:33.106997 9903 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:33.108376 9828 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:33.116999 9903 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.118990 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:32.969899 12688 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:33.119225 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 100818us
I20260430 07:59:33.119640 10004 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 9.
I20260430 07:59:33.093382 12410 raft_consensus.cc:2749] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:33.166836 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 47000.6us
I20260430 07:59:33.034312 12685 consensus_queue.cc:1048] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.080s
I20260430 07:59:33.121085 9903 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:59:33.167775 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 47274us
I20260430 07:59:33.189855 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:33.166126 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.042975 12614 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.174476 12749 consensus_queue.cc:1059] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:33.166857 12453 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:33.102751 12648 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:33.186261 12716 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:32.997984 12737 consensus_queue.cc:237] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:33.174562 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:33.169397 12776 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:33.191663 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 70662.4us
I20260430 07:59:33.190845 9904 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:33.219408 9904 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.120695 12619 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:33.095948 12782 raft_consensus.cc:2804] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:33.024470 12754 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:33.022486 12662 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:33.198361 9863 raft_consensus.cc:2410] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260430 07:59:33.120607 12822 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260430 07:59:33.129146 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:33.126015 12640 raft_consensus.cc:2749] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:33.249197 10004 raft_consensus.cc:1240] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:33.249584 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.206547 12716 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:33.233915 9860 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 10.
I20260430 07:59:33.198189 12766 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:33.241021 9904 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:33.241973 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 46606.1us
W20260430 07:59:33.239523 9977 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.509s (SENT) [suppressed 2 similar messages]
I20260430 07:59:33.242002 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:33.115898 12797 consensus_queue.cc:1048] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.105s
I20260430 07:59:33.257012 12782 raft_consensus.cc:697] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 7 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:33.283354 12617 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:33.247822 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 75132.3us
I20260430 07:59:33.206302 12479 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:33.289208 12479 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.289991 12743 raft_consensus.cc:3055] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:33.290426 12743 raft_consensus.cc:740] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:33.291224 12743 consensus_queue.cc:260] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.291950 12479 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:33.222261 12749 consensus_queue.cc:1243] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer 48e81934d52a4e93be5171ceea193032 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260430 07:59:33.292294 12743 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:33.095476 12779 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:33.232420 12619 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:33.293848 12619 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:33.222749 12422 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:33.238960 12410 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:33.302003 12422 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.302268 12410 raft_consensus.cc:697] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:33.303725 12410 consensus_queue.cc:237] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.241209 12764 raft_consensus.cc:2804] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260430 07:59:33.243261 12524 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:33.305454 12764 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:33.305938 12764 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 2 FOLLOWER]: Advancing to term 3
I20260430 07:59:33.254451 12716 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:33.307114 12301 consensus_queue.cc:1048] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:59:33.269367 9978 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:33.243417 12754 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:33.278646 9856 raft_consensus.cc:1275] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260430 07:59:33.314987 9856 pending_rounds.cc:85] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 3
I20260430 07:59:33.315323 9856 pending_rounds.cc:107] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260430 07:59:33.315637 9856 raft_consensus.cc:2889] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:33.283025 12782 consensus_queue.cc:237] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:33.288935 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:33.211400 12791 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:33.304807 12422 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:33.305631 12524 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.277994 9904 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:33.202940 12784 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:33.314465 12721 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:33.333187 12784 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:33.335376 9828 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.335734 12784 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:33.247404 12822 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:33.250639 12662 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.329034 9978 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:33.330363 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 52429.9us
I20260430 07:59:33.309882 12758 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:33.328159 12791 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.348730 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.349514 10004 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:33.350497 12791 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:33.352133 12709 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.354158 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:33.330655 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:33.355762 10559 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:33.341912 12721 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:33.357601 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:33.358029 12662 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.361002 12797 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:33.361685 9904 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:33.362077 12839 consensus_queue.cc:1059] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:33.363015 12812 consensus_queue.cc:1059] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:33.364022 12812 raft_consensus.cc:3055] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:33.364384 12812 raft_consensus.cc:740] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:33.364751 12839 raft_consensus.cc:3055] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:33.317800 12799 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:33.317703 12754 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:33.377260 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.318277 12798 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 07:59:33.386659 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.372465 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.384702 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:59:33.387573 10004 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:33.390170 12758 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:33.390643 12758 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:33.391474 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.394034 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.415174 10010 raft_consensus.cc:2393] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:33.395936 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.399758 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.417734 10004 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
I20260430 07:59:33.397324 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.412163 9904 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:33.399041 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.388489 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:59:33.420035 10003 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:33.392498 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 153189us
I20260430 07:59:33.430526 9904 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:33.430584 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.435554 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.437165 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.439298 9855 raft_consensus.cc:1240] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:33.442606 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.369159 12822 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:33.343756 12794 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:33.446278 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:33.365123 12839 raft_consensus.cc:740] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:33.447367 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:33.455147 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 172930us
I20260430 07:59:33.456269 10012 raft_consensus.cc:1240] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:33.412510 12617 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:33.455904 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:33.369431 10560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54770 (request call id 390) took 2613 ms. Trace:
I20260430 07:59:33.367851 12422 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Leader election won for term 10
I20260430 07:59:33.449882 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.451458 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.466239 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.467094 9930 raft_consensus.cc:2468] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
I20260430 07:59:33.452283 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.447507 12797 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:33.390992 12758 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:33.451357 12794 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:33.457701 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:33.458334 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.479526 9936 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:33.480433 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.368927 12479 consensus_queue.cc:1048] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20260430 07:59:33.391396 12798 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:33.455956 9828 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:33.458504 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.462178 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.462370 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.483150 9937 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
I20260430 07:59:33.483327 10012 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:33.464545 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.447350 12766 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:33.450093 9977 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.486203 9977 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.487309 9977 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:33.468566 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.488158 10004 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:33.470497 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.470690 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.451228 12640 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:33.451205 12618 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:33.468801 12770 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 07:59:33.391598 12799 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.476843 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.483388 9828 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:33.483350 9931 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:33.451813 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.495929 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.365218 12812 consensus_queue.cc:260] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.460821 12256 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:33.484090 12794 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:33.490581 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.464761 12617 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.457017 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.491789 9930 raft_consensus.cc:1275] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:33.492707 10007 raft_consensus.cc:1275] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:33.471055 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:33.476096 12422 raft_consensus.cc:697] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 10 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:33.520185 12422 consensus_queue.cc:237] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.521777 12822 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:33.523146 10566 raft_consensus.cc:1240] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:33.527415 12822 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.497812 12618 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.531473 12618 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.532294 9935 raft_consensus.cc:1275] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:33.498610 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.489883 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:33.502288 9828 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.535135 9933 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:33.471676 12722 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:33.489149 12798 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:33.486129 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:33.490159 12566 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:33.540958 12566 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:33.463212 10560 rpcz_store.cc:276] 0430 07:59:30.755530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:30.758744 (+ 3214us) service_pool.cc:225] Handling call
0430 07:59:30.768447 (+ 9703us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:30.970748 (+202301us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:30.976355 (+ 5607us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:32.787342 (+1810987us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b2c00007668. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb567b7dc81 00007fb567b8d5e2 00007fb5692bede1 00007fb56928381a 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22
0430 07:59:32.787982 (+ 640us) log.cc:844] Serialized 52 byte log entry
0430 07:59:32.947819 (+159837us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:33.201853 (+254034us) log.cc:844] Serialized 14 byte log entry
0430 07:59:33.255484 (+ 53631us) log.cc:844] Serialized 14 byte log entry
0430 07:59:33.269985 (+ 14501us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:33.316077 (+ 46092us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:33.338051 (+ 21974us) raft_consensus.cc:1685] finished
0430 07:59:33.338075 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:33.369337 (+ 31262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26546,"prepare.queue_time_us":434451,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":4098,"raft.queue_time_us":267564,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":9725,"spinlock_wait_cycles":33434368,"thread_start_us":583961,"threads_started":3,"wal-append.queue_time_us":174180}
I20260430 07:59:33.541476 12566 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:33.507930 12685 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:33.542090 12685 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:33.542745 12685 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:33.543959 12721 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:33.510288 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:33.493775 12737 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:33.511904 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.499934 12815 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:33.514081 10007 raft_consensus.cc:1275] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:33.482158 12619 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.490814 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.496912 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:33.492827 12770 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:33.590693 12770 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Advancing to term 10
W20260430 07:59:33.591213 12739 consensus_meta.cc:287] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.590s user 0.010s sys 0.000s
I20260430 07:59:33.591764 12739 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.486574 12797 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.503664 12791 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 07:59:33.600008 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:33.541257 9828 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:59:33.601622 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 60439.1us
I20260430 07:59:33.602833 9828 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.604017 9828 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:33.606837 9930 raft_consensus.cc:1240] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260430 07:59:33.619508 10012 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:33.544889 9856 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:33.511210 9937 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 9. Current term is 10. Ops: []
I20260430 07:59:33.514391 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.624913 10004 raft_consensus.cc:2393] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 9: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:33.514070 10010 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 10 LEADER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 9. Current term is 10. Ops: [7.5-9.6]
I20260430 07:59:33.497675 12640 raft_consensus.cc:697] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:33.492439 12622 consensus_queue.cc:1059] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260430 07:59:33.499882 12766 raft_consensus.cc:697] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:33.515116 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.605005 9828 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:33.556566 12524 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.557587 12617 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.558984 12799 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.550834 12301 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:33.564025 12798 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.564805 12812 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:33.564951 12714 raft_consensus.cc:3055] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Stepping down as leader of term 9
I20260430 07:59:33.504299 12256 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.565794 12709 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:33.567036 12685 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.573241 12453 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:33.497905 12839 consensus_queue.cc:260] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.578589 12722 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:33.579180 9757 catalog_manager.cc:5671] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:59:33.580440 12716 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:33.595214 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 58385.9us
I20260430 07:59:33.597527 9978 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:33.622215 10008 raft_consensus.cc:1240] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260430 07:59:33.628039 9930 raft_consensus.cc:1275] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260430 07:59:33.705874 12739 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.543879 12764 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:33.775071 12764 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:33.600297 12791 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:33.673774 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.775903 12791 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:33.684338 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:33.686409 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 468175us
I20260430 07:59:33.778311 9903 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:33.610948 12773 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:33.779736 9903 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.779888 12410 raft_consensus.cc:2749] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:33.687875 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:33.691197 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:33.672326 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.703599 9828 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:33.782048 12776 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:33.782287 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 78746.4us
I20260430 07:59:33.782953 9828 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:33.783027 12640 consensus_queue.cc:237] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.785233 12301 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:33.785799 12301 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:33.786335 12301 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:33.723117 9856 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:33.788725 12524 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:33.789258 12524 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:33.789712 12524 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:33.616212 12451 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:33.790994 12664 consensus_queue.cc:1048] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:33.623202 12678 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:33.792250 12678 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:33.792761 12678 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:33.721956 12714 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:33.741751 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:33.692420 12622 raft_consensus.cc:3055] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:33.795482 12622 raft_consensus.cc:740] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:33.698951 12670 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:33.690402 12798 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:33.796037 12670 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:33.702291 12619 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.615561 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.796487 12670 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:33.703290 12797 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.796355 12622 consensus_queue.cc:260] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:33.732345 12453 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:33.769958 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 172527us
I20260430 07:59:33.770644 12707 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:33.818149 12707 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:33.818506 9862 raft_consensus.cc:1275] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260430 07:59:33.705463 10005 raft_consensus.cc:2393] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 10: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:33.806278 9857 raft_consensus.cc:1275] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:33.637538 12479 consensus_queue.cc:1059] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.115s
I20260430 07:59:33.807024 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:33.820976 9856 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:33.818609 12707 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:33.823742 9978 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:33.825348 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 45683.7us
I20260430 07:59:33.829252 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:33.840663 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:33.849900 9860 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260430 07:59:33.850368 9857 raft_consensus.cc:2393] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 9: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:33.854449 9903 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:33.855804 9903 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:59:33.869551 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 45917.8us
I20260430 07:59:33.872010 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:33.873822 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.875005 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.875644 10008 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:33.878223 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.881235 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:33.880838 10008 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:33.879794 9903 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:33.879024 9856 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 10.
I20260430 07:59:33.884418 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.884822 9930 raft_consensus.cc:2393] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 9: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:33.885309 9933 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:33.886052 9978 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:33.759699 12617 consensus_queue.cc:1059] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260430 07:59:33.699781 12766 consensus_queue.cc:237] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.896886 9903 leader_election.cc:400] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 8. Current term is 9.
I20260430 07:59:33.897250 9903 leader_election.cc:403] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260430 07:59:33.890642 10533 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:33.893141 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:33.888482 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.899116 10007 raft_consensus.cc:3060] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:33.901160 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:33.879763 12369 raft_consensus.cc:2804] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election won for term 9
I20260430 07:59:33.906246 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.875330 12618 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:33.907585 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.888054 12764 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:33.909049 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:33.911835 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.919844 10006 raft_consensus.cc:2393] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:33.912047 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.884048 12422 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election won for term 9
I20260430 07:59:33.922624 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:33.923702 9863 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:33.920208 10009 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:33.882817 12685 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:33.911587 9977 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:33.875890 12619 consensus_queue.cc:1059] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260430 07:59:33.908305 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.915580 12654 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:33.890262 12483 consensus_queue.cc:1048] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260430 07:59:33.921510 9828 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.926474 9978 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:33.860064 12689 consensus_queue.cc:1048] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.043s
I20260430 07:59:33.903918 12479 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Leader election won for term 10
I20260430 07:59:33.927992 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.947062 10011 raft_consensus.cc:2410] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260430 07:59:33.916867 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.948487 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.949293 10010 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 9.
I20260430 07:59:33.950543 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.933586 10007 raft_consensus.cc:2468] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
I20260430 07:59:33.951864 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.952550 10007 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:33.930101 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.904280 12410 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:33.901351 12831 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:33.920732 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.962642 10006 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 7: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:33.939939 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.794746 12714 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.954838 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:33.968453 10533 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:33.959259 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:33.924391 12451 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:33.924788 12799 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 9
I20260430 07:59:33.971745 9828 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:33.901687 12794 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.939082 12648 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:33.976038 10011 raft_consensus.cc:2410] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260430 07:59:33.975824 9828 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:33.942900 12422 raft_consensus.cc:697] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [term 9 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:33.987042 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:33.988111 10003 raft_consensus.cc:2393] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 9: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:33.922768 12618 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.957070 12256 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:33.967358 12815 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:33.998296 9938 raft_consensus.cc:2393] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:33.978962 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:33.961239 12685 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:34.000926 12685 raft_consensus.cc:2764] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
W20260430 07:59:34.001379 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 98706.2us
I20260430 07:59:34.002558 9828 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:34.003343 12451 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:34.003990 12451 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:34.004465 12451 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:33.977484 12450 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:33.968048 12479 raft_consensus.cc:697] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 10 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:34.010040 9828 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:33.952050 12654 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:33.980343 12619 raft_consensus.cc:2764] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260430 07:59:33.986218 12772 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:33.987919 12788 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:33.891317 12783 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.036s
I20260430 07:59:34.057262 9828 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:34.060554 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:34.066993 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:33.897632 12737 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
W20260430 07:59:34.022243 12754 consensus_meta.cc:287] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.592s user 0.007s sys 0.004s
I20260430 07:59:34.073112 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.074779 9933 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:34.006943 12770 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:34.055770 12758 consensus_meta.cc:287] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.569s user 0.008s sys 0.002s
I20260430 07:59:34.011484 12566 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.012921 12369 raft_consensus.cc:697] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 9 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:34.077900 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:34.078367 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.194156 12369 consensus_queue.cc:237] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.195071 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.195933 9933 raft_consensus.cc:2393] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:34.196813 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.015185 12772 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:34.018069 12716 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:34.068261 12749 consensus_queue.cc:1048] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.067s
I20260430 07:59:34.205626 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:34.205951 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.019909 12298 consensus_queue.cc:1059] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.247s
I20260430 07:59:34.081073 12689 raft_consensus.cc:2764] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
W20260430 07:59:34.136739 12851 log.cc:927] Time spent T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.162s user 0.000s sys 0.001s
W20260430 07:59:34.137689 12855 log.cc:927] Time spent T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
W20260430 07:59:34.138259 12853 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.203s user 0.000s sys 0.001s
I20260430 07:59:34.080850 12754 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.142495 12794 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.086315 12450 consensus_queue.cc:1048] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.060s
I20260430 07:59:34.145510 12685 consensus_queue.cc:1059] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:34.150327 12783 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:34.154467 12758 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.058422 12619 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:34.154736 12479 consensus_queue.cc:237] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:33.976737 12797 consensus_queue.cc:1175] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "48e81934d52a4e93be5171ceea193032" responder_term: 10 status { last_received { term: 10 index: 6 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 9. Current term is 10. Ops: [7.5-9.6]" } } last_received_current_leader { term: 0 index: 0 } }
W20260430 07:59:34.165488 12856 log.cc:927] Time spent T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.191s user 0.000s sys 0.001s
I20260430 07:59:34.169366 12422 consensus_queue.cc:237] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.170866 12654 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.172942 12770 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 10 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:34.439241 12524 consensus_meta.cc:287] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.649s user 0.010s sys 0.003s
I20260430 07:59:34.177034 12779 raft_consensus.cc:3055] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:34.439675 12524 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.179373 12815 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:34.440793 12815 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:34.186933 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 101464us
W20260430 07:59:34.443212 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:34.195160 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:34.080955 9863 raft_consensus.cc:1275] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260430 07:59:34.187144 12566 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.135259 12857 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:34.445981 12857 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.446579 9862 raft_consensus.cc:2393] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 8: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:34.448486 12857 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:34.449604 12670 consensus_meta.cc:287] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.652s user 0.016s sys 0.003s
I20260430 07:59:34.450017 12670 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.451370 12451 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.452500 9857 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260430 07:59:34.210565 10533 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:34.213416 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.455008 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.456318 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:34.233376 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:34.421926 12773 consensus_meta.cc:287] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.640s user 0.010s sys 0.002s
I20260430 07:59:34.457687 12773 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:34.421255 10559 raft_consensus.cc:1240] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:34.201960 12772 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:34.316833 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:34.460152 12622 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:34.199684 10568 raft_consensus.cc:1240] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 10 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 9. Current term is 10. Ops: []
W20260430 07:59:34.399443 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 259354us
W20260430 07:59:34.360122 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:34.464793 9854 raft_consensus.cc:1240] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:34.154541 12618 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.465724 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.466185 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:34.467171 12814 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:34.467705 12814 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.468055 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:34.468355 12812 consensus_meta.cc:287] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.750s user 0.008s sys 0.002s
I20260430 07:59:34.468698 10012 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 8
I20260430 07:59:34.469442 12709 consensus_queue.cc:1048] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:59:34.469245 10566 consensus_queue.cc:237] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:34.469910 12814 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:34.423630 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:34.427632 12410 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:34.427910 12678 consensus_meta.cc:287] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.635s user 0.012s sys 0.000s
I20260430 07:59:34.431696 12758 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.433023 9757 catalog_manager.cc:5671] T db24b74e2fa44ae2beba675582424a3b P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260430 07:59:34.436203 12707 consensus_meta.cc:287] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.570s user 0.010s sys 0.000s
W20260430 07:59:34.438820 12301 consensus_meta.cc:287] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.652s user 0.011s sys 0.003s
I20260430 07:59:34.439802 12779 raft_consensus.cc:740] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:34.444905 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:34.454955 12759 raft_consensus.cc:3055] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:34.555366 12759 raft_consensus.cc:740] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:34.556865 12759 consensus_queue.cc:260] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.470693 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:34.472836 9856 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: []
W20260430 07:59:34.414444 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 332851us
I20260430 07:59:34.559568 12759 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:34.560554 9828 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:34.411187 9932 raft_consensus.cc:2368] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:34.474126 10011 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:34.474896 9977 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:34.475793 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.479977 9903 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:34.571035 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 91235.3us
W20260430 07:59:34.409178 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:34.486460 10012 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:34.478112 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:34.520416 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:34.452209 12670 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.524835 9937 raft_consensus.cc:2393] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:34.582444 10004 raft_consensus.cc:2410] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260430 07:59:34.519857 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.583999 10003 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 8
I20260430 07:59:34.588702 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.589489 9933 raft_consensus.cc:2368] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8.
W20260430 07:59:34.489089 12764 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.002s sys 0.012s
I20260430 07:59:34.524024 12301 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.527767 12451 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.527863 12754 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.528903 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
W20260430 07:59:34.688235 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 46643.7us
W20260430 07:59:34.531121 12791 consensus_meta.cc:287] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.755s user 0.011s sys 0.005s
I20260430 07:59:34.534020 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1169) took 1023 ms. Trace:
I20260430 07:59:34.416522 12783 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:34.521617 12799 consensus_meta.cc:287] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.539s user 0.010s sys 0.000s
W20260430 07:59:34.539168 12453 consensus_meta.cc:287] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.721s user 0.016s sys 0.003s
I20260430 07:59:34.551200 12524 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.495393 12678 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.553655 12779 consensus_queue.cc:260] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:34.739934 12678 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.576574 9903 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:34.592767 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.741061 9859 raft_consensus.cc:1240] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
W20260430 07:59:34.595714 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 120880us
I20260430 07:59:34.743244 9977 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:34.743914 12783 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.744624 12783 raft_consensus.cc:2749] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260430 07:59:34.644817 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:34.745994 12524 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:34.746613 12524 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.674122 12709 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:34.751364 12709 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.753885 12453 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.756225 12453 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.757125 9858 raft_consensus.cc:1240] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:34.730177 12799 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 8. Current term is 9.
I20260430 07:59:34.744901 9977 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:34.763958 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 190649us
I20260430 07:59:34.764874 9903 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:34.766045 9903 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:34.767099 9903 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:34.640738 12867 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:34.681522 9828 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:34.773150 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 91753.4us
I20260430 07:59:34.774179 9828 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:34.524363 12707 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:34.657768 12764 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s user 0.003s sys 0.013s
W20260430 07:59:34.666523 12617 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.009s sys 0.009s
I20260430 07:59:34.710042 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:34.692409 9935 rpcz_store.cc:276] 0430 07:59:33.582065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.582205 (+ 140us) service_pool.cc:225] Handling call
0430 07:59:33.582548 (+ 343us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:33.582899 (+ 351us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:33.582927 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:33.599150 (+ 16223us) log.cc:844] Serialized 52 byte log entry
0430 07:59:33.869056 (+269906us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:33.916372 (+ 47316us) log.cc:844] Serialized 14 byte log entry
0430 07:59:33.997463 (+ 81091us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:33.997538 (+ 75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:34.533770 (+536232us) raft_consensus.cc:1685] finished
0430 07:59:34.533794 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:34.533955 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11451,"prepare.queue_time_us":53447,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":4722,"raft.queue_time_us":61041,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":12270,"thread_start_us":261096,"threads_started":2,"wal-append.queue_time_us":349179}
I20260430 07:59:34.641999 12749 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:34.692415 12865 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:34.728315 12791 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:34.673195 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.596246 12868 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:34.768886 10003 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:34.688501 12870 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:34.800485 12617 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.875s user 0.010s sys 0.009s
I20260430 07:59:34.786463 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.795310 12566 raft_consensus.cc:2804] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election won for term 9
I20260430 07:59:34.855543 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.854027 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.855064 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.853015 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:34.865326 10009 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:34.865581 10004 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:34.864042 10012 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:34.844328 10007 raft_consensus.cc:1240] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 9. Current term is 10. Ops: []
I20260430 07:59:34.787374 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1168) took 1281 ms. Trace:
I20260430 07:59:34.587971 12812 consensus_queue.cc:1048] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.141s
I20260430 07:59:34.800865 9937 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:34.802168 10005 raft_consensus.cc:1275] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260430 07:59:34.803122 12618 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:34.761745 12524 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.817555 9933 raft_consensus.cc:2393] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:34.807508 12867 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:34.794020 12422 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:34.821664 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.790374 12453 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:34.792802 12758 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:34.783001 9903 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:34.870097 9931 rpcz_store.cc:276] 0430 07:59:33.506248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.506406 (+ 158us) service_pool.cc:225] Handling call
0430 07:59:33.506667 (+ 261us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:33.506858 (+ 191us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:33.641240 (+134382us) log.cc:844] Serialized 14 byte log entry
0430 07:59:34.478942 (+837702us) spinlock_profiling.cc:243] Waited 8.03 ms on lock 0x7b18004318c0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb568e2a85c 00007fb5661c8c0b 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e5242 00007fb568e1c5ab 00007fb568e220c3 00007fb568e225da
0430 07:59:34.769778 (+290836us) log.cc:844] Serialized 14 byte log entry
0430 07:59:34.784271 (+ 14493us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:34.784361 (+ 90us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:34.784432 (+ 71us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:34.784536 (+ 104us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:34.787306 (+ 2770us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58911,"prepare.queue_time_us":252646,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":8421,"raft.queue_time_us":47860,"raft.run_cpu_time_us":483,"raft.run_wall_time_us":64372,"spinlock_wait_cycles":18458368,"thread_start_us":619469,"threads_started":3,"wal-append.queue_time_us":286270}
I20260430 07:59:34.874464 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.896938 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.897332 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.795559 12754 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 07:59:34.675092 12772 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.875424 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.795764 12764 raft_consensus.cc:697] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:34.853423 12737 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:34.961843 12737 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.962180 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:34.962015 12764 consensus_queue.cc:237] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:34.964098 10009 raft_consensus.cc:2393] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:34.805879 12749 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:34.965500 12749 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:34.966969 12772 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.883623 9930 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:34.888358 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:34.969760 10005 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:34.889539 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.892299 9828 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:34.971769 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:34.895977 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:34.897848 9931 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:34.973832 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.975184 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:34.976394 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:34.879760 12856 log.cc:927] Time spent T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
W20260430 07:59:34.914629 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:34.920426 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:34.924482 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:34.965512 9862 raft_consensus.cc:1240] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:34.879251 12618 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:34.875411 12857 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:34.889071 12709 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:34.891973 12566 raft_consensus.cc:697] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:34.997114 12618 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:34.844034 12791 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:34.897992 10010 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:35.009992 12857 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.012522 12857 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.896050 12453 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:35.013054 12566 consensus_queue.cc:237] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.013731 12453 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:34.819015 12865 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:34.998883 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:34.935981 10004 raft_consensus.cc:2468] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
W20260430 07:59:34.990252 12871 log.cc:927] Time spent T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.178s user 0.000s sys 0.000s
W20260430 07:59:34.999585 12874 log.cc:927] Time spent T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260430 07:59:35.006310 12758 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:34.992352 10011 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:35.006289 12707 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:35.004601 12617 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:35.018582 12617 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.019524 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:35.020255 9935 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:35.021214 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:34.979686 12689 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:35.022884 12689 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.024609 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:34.863996 12815 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.025964 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:35.016222 10010 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:35.027189 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:35.024963 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.027663 12815 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:35.028362 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:35.028638 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 136440us
I20260430 07:59:35.025022 12689 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.029716 12689 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:35.030086 12689 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:35.030479 12689 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:35.032555 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:34.902225 12754 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:34.993676 10012 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:35.034026 12754 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:34.845118 12301 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:34.892412 12524 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:35.034868 12524 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.036269 9937 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:34.891507 12422 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:35.036854 12524 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.037062 12422 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:35.037256 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:35.037969 9937 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:34.998245 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:35.038655 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:34.874131 12870 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.021011 12617 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.041211 9861 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:35.006083 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 223283us
I20260430 07:59:35.054539 9903 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:35.042176 9931 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:35.049886 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 320555us
W20260430 07:59:35.039673 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.038972 9860 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:35.016727 12865 leader_election.cc:290] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:35.057403 9828 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:35.055749 12670 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:35.067795 12670 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.070016 12670 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.071802 9933 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:35.073007 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1604) took 1004 ms. Trace:
I20260430 07:59:35.073508 9933 rpcz_store.cc:276] 0430 07:59:34.069096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:34.971310 (+902214us) service_pool.cc:225] Handling call
0430 07:59:35.072829 (+101519us) spinlock_profiling.cc:243] Waited 99.1 ms on lock 0x7b54001b2bd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:35.072944 (+ 115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":227989760}
I20260430 07:59:35.074975 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:35.060089 12766 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:35.061849 9936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1162) took 1591 ms (client timeout 1736 ms). Trace:
W20260430 07:59:35.082918 9936 rpcz_store.cc:269] 0430 07:59:33.470008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.480131 (+ 10123us) service_pool.cc:225] Handling call
0430 07:59:35.061645 (+1581514us) spinlock_profiling.cc:243] Waited 731 ms on lock 0x7b54001aca54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:35.061787 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1680495360}
I20260430 07:59:35.084573 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:35.065016 10560 raft_consensus.cc:1240] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [term 7 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: []
W20260430 07:59:35.148567 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.065548 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:35.002964 12878 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.231s user 0.000s sys 0.000s
I20260430 07:59:35.065734 12524 consensus_queue.cc:1048] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260430 07:59:35.003604 12875 log.cc:927] Time spent T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
I20260430 07:59:35.036787 12618 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.000385 12839 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:34.999383 12749 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.076140 9828 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:35.172842 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 96773us
I20260430 07:59:35.173784 9828 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:35.076675 12797 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:35.059212 9903 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:35.065433 12737 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:35.052194 12857 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:35.077474 12870 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:35.080664 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:35.088442 9931 raft_consensus.cc:2468] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:35.091074 9936 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Advancing to term 10
W20260430 07:59:35.100534 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 355565us
I20260430 07:59:35.149389 9935 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:35.154547 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1859) took 1247 ms. Trace:
I20260430 07:59:35.151695 12867 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:35.155514 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1842) took 1313 ms. Trace:
I20260430 07:59:35.154598 12453 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.205952 12857 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.178045 12685 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:35.208238 12685 raft_consensus.cc:697] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:35.079690 12422 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.209342 12685 consensus_queue.cc:237] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.158088 12788 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:35.211150 12422 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:35.211573 12788 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.213423 10006 raft_consensus.cc:1240] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:35.213661 12788 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.167738 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1589) took 1107 ms. Trace:
I20260430 07:59:35.214337 9930 rpcz_store.cc:276] 0430 07:59:34.060129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:34.060276 (+ 147us) service_pool.cc:225] Handling call
0430 07:59:35.167370 (+1107094us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b540036c9d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:35.167531 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1610382720}
I20260430 07:59:35.215462 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:35.215857 12749 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.216109 9930 raft_consensus.cc:2376] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9. Re-sending same reply.
I20260430 07:59:35.216696 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1596) took 1153 ms. Trace:
I20260430 07:59:35.180234 9828 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:35.217926 9828 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:35.218220 12870 raft_consensus.cc:2749] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:35.196005 12670 raft_consensus.cc:2804] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:35.196187 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:35.219874 12670 raft_consensus.cc:697] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:35.220546 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:35.221356 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1614) took 1085 ms. Trace:
I20260430 07:59:35.221148 12670 consensus_queue.cc:237] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.222484 12618 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.222712 12759 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:35.223227 12759 raft_consensus.cc:697] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:35.224123 12759 consensus_queue.cc:237] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.151475 12743 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:35.226433 12743 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.228402 12743 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:35.154917 10560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54770 (request call id 407) took 1234 ms. Trace:
I20260430 07:59:35.229135 10560 rpcz_store.cc:276] 0430 07:59:33.920953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.923087 (+ 2134us) service_pool.cc:225] Handling call
0430 07:59:35.064580 (+1141493us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b58001f1798. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:59:35.064820 (+ 240us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:35.151370 (+ 86550us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.154857 (+ 3487us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30515456}
I20260430 07:59:35.230345 10567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:48088 (request call id 11) took 1158 ms. Trace:
I20260430 07:59:35.230762 10567 rpcz_store.cc:276] 0430 07:59:34.071805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:34.072009 (+ 204us) service_pool.cc:225] Handling call
0430 07:59:35.229792 (+1157783us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b540063c4d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ed470 00007fb56928c31a 00007fb56e25b11e 00007fb5677d7437 00007fb5677d73c2 00007fb5677d72f1 00007fb5677d726c 00007fb5677d6522 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:35.230262 (+ 470us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":795442944}
I20260430 07:59:35.231278 12867 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.235962 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1845) took 1156 ms. Trace:
I20260430 07:59:35.196578 10007 rpcz_store.cc:276] 0430 07:59:33.907488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.964743 (+ 57255us) service_pool.cc:225] Handling call
0430 07:59:33.981537 (+ 16794us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:34.868132 (+886595us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.154285 (+286153us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b54000f7dd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:35.154484 (+ 199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1983486208}
I20260430 07:59:35.236419 9856 rpcz_store.cc:276] 0430 07:59:34.079441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:34.079662 (+ 221us) service_pool.cc:225] Handling call
0430 07:59:34.080011 (+ 349us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:34.558635 (+478624us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.235736 (+677101us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b5400207e54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:35.235902 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":902798080}
I20260430 07:59:35.201927 9937 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:35.176987 12797 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:35.237676 12797 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:35.237735 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:35.239161 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1594) took 1176 ms. Trace:
I20260430 07:59:35.239509 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:35.240721 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:35.200640 9859 rpcz_store.cc:276] 0430 07:59:33.842501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.842693 (+ 192us) service_pool.cc:225] Handling call
0430 07:59:33.843085 (+ 392us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:34.741622 (+898537us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.155304 (+413682us) spinlock_profiling.cc:243] Waited 898 ms on lock 0x7b54001ce3d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:35.155462 (+ 158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2064276352}
W20260430 07:59:35.202998 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.219558 9828 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:35.167382 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 549) took 1339 ms. Trace:
I20260430 07:59:35.206065 12772 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:35.208304 12453 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.221760 9931 rpcz_store.cc:276] 0430 07:59:34.135794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.220258 (+1084464us) service_pool.cc:225] Handling call
0430 07:59:35.221301 (+ 1043us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:35.255065 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:35.204145 9977 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:35.243068 9854 rpcz_store.cc:276] 0430 07:59:33.827877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.828636 (+ 759us) service_pool.cc:225] Handling call
0430 07:59:33.869169 (+ 40533us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:34.465258 (+596089us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.167030 (+701772us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b54001bab54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:35.167304 (+ 274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1369117568}
I20260430 07:59:35.244395 9828 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:35.255553 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:35.256265 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:35.216992 9930 rpcz_store.cc:276] 0430 07:59:34.063667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.215188 (+1151521us) service_pool.cc:225] Handling call
0430 07:59:35.216638 (+ 1450us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:35.245994 9861 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
W20260430 07:59:35.201126 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 141981us
I20260430 07:59:35.257570 9903 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:35.258931 12799 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:35.265638 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:35.266703 10012 raft_consensus.cc:2393] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:35.266752 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:35.267520 9859 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:35.149372 12689 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.239539 9937 rpcz_store.cc:276] 0430 07:59:34.063214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.042230 (+979016us) service_pool.cc:225] Handling call
0430 07:59:35.239007 (+196777us) spinlock_profiling.cc:243] Waited 7.2 ms on lock 0x7b540036c4d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:35.239111 (+ 104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16548736}
I20260430 07:59:35.256857 9856 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:35.236173 12754 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.268009 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:35.279640 10011 raft_consensus.cc:2393] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 7: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:35.265595 9903 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:35.282351 9904 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:35.259860 12301 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260430 07:59:35.270952 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:35.304740 10009 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 8: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:35.270823 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:35.306464 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:35.307456 9930 raft_consensus.cc:2393] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:35.308589 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:35.271508 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 67481.4us
I20260430 07:59:35.271776 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:35.272701 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:35.272843 9936 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 10.
I20260430 07:59:35.222359 12812 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:35.275815 10534 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:35.277084 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:35.283565 9903 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:35.265472 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:35.284130 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:35.284387 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:35.284363 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:35.284667 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:35.311132 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1623) took 1172 ms. Trace:
I20260430 07:59:35.311141 9931 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:35.311468 9977 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:35.270447 12619 raft_consensus.cc:2804] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:35.313656 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:35.338033 9937 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:35.338871 10005 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:35.218259 12857 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:35.339709 9933 raft_consensus.cc:2393] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 10: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:35.340054 10004 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:35.273885 12872 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:35.275329 12743 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:35.341569 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:35.339725 9936 rpcz_store.cc:276] 0430 07:59:34.138723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.084249 (+945526us) service_pool.cc:225] Handling call
0430 07:59:35.311072 (+226823us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:35.344723 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:35.344731 9977 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:35.345654 9936 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:35.346839 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:35.347498 9936 raft_consensus.cc:3060] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:35.347261 9977 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:35.348881 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 250278us
I20260430 07:59:35.260011 12617 consensus_queue.cc:1059] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260430 07:59:35.361555 9935 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
I20260430 07:59:35.302958 12410 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:35.367588 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 91857.6us
I20260430 07:59:35.284119 12453 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:35.309492 12799 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:35.260815 12870 consensus_queue.cc:1059] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
I20260430 07:59:35.270623 12772 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.341377 12764 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:35.363999 12857 raft_consensus.cc:2749] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:35.343508 12867 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:35.362618 12422 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:35.367053 12619 raft_consensus.cc:697] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:35.372108 12737 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:35.378002 12788 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:35.378444 12812 raft_consensus.cc:697] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:35.349574 12872 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.377707 12453 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:35.615713 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.379696 9977 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:35.463313 12689 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.620311 12256 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:35.589690 9856 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 6.
I20260430 07:59:35.588788 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:35.465682 12872 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.642494 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:35.643859 9854 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:35.395232 12737 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.533944 12721 raft_consensus.cc:3055] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:35.467579 12866 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:35.519467 12772 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:35.652084 12866 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.656227 12776 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:35.657403 12776 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.659955 12868 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:35.660633 12776 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:35.660627 12857 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:35.660948 12868 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.661938 12857 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.664252 12868 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:35.393487 12788 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:35.671000 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.460834 12619 consensus_queue.cc:237] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.676977 12256 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.678826 12422 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:35.679281 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1867) took 1494 ms. Trace:
I20260430 07:59:35.679637 12422 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.680292 12256 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:35.598307 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:35.681614 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1859) took 1878 ms. Trace:
I20260430 07:59:35.682770 9859 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:35.682368 9858 rpcz_store.cc:276] 0430 07:59:33.803318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.803438 (+ 120us) service_pool.cc:225] Handling call
0430 07:59:33.818042 (+ 14604us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:34.757628 (+939586us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.680836 (+923208us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b54003bf6d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:35.681291 (+ 455us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2158810368}
I20260430 07:59:35.670892 12788 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:35.692253 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1873) took 1153 ms. Trace:
I20260430 07:59:35.694456 12422 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:35.694806 9929 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1151) took 2243 ms (client timeout 1902 ms). Trace:
I20260430 07:59:35.696769 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1595) took 1633 ms. Trace:
W20260430 07:59:35.698405 9932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1152) took 2246 ms (client timeout 1811 ms). Trace:
I20260430 07:59:35.699862 12737 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:35.700366 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1150) took 2250 ms. Trace:
I20260430 07:59:35.540355 12799 raft_consensus.cc:697] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [term 8 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:35.702279 9931 raft_consensus.cc:1240] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 7. Ops: []
I20260430 07:59:35.711078 12453 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:35.711548 12721 raft_consensus.cc:740] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:35.713048 9936 raft_consensus.cc:2468] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:35.713639 12721 consensus_queue.cc:260] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:35.707566 9929 rpcz_store.cc:269] 0430 07:59:33.450905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.451585 (+ 680us) service_pool.cc:225] Handling call
0430 07:59:35.694152 (+2242567us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540036d3d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:35.694494 (+ 342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2324076416}
I20260430 07:59:35.679988 9857 rpcz_store.cc:276] 0430 07:59:34.184958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:34.192913 (+ 7955us) service_pool.cc:225] Handling call
0430 07:59:34.197025 (+ 4112us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:34.452891 (+255866us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.678361 (+1225470us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b540029e854. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:35.678991 (+ 630us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":586901248}
W20260430 07:59:35.704073 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 338314us
I20260430 07:59:35.710290 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 11 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:35.708061 9932 rpcz_store.cc:269] 0430 07:59:33.451900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.452059 (+ 159us) service_pool.cc:225] Handling call
0430 07:59:35.697864 (+2245805us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b54001ee954. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:35.698158 (+ 294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2147045120}
I20260430 07:59:35.462077 12754 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:35.730067 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.387742 12764 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.523618 12797 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.466377 12812 consensus_queue.cc:237] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:35.732318 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:35.729535 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.707633 9933 rpcz_store.cc:276] 0430 07:59:34.063491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.074693 (+1011202us) service_pool.cc:225] Handling call
0430 07:59:35.696227 (+621534us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b54003232d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:35.696536 (+ 309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":596103936}
I20260430 07:59:35.706295 9862 rpcz_store.cc:276] 0430 07:59:34.538372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:34.538519 (+ 147us) service_pool.cc:225] Handling call
0430 07:59:34.538903 (+ 384us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:34.979696 (+440793us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.691391 (+711695us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b54001ba654. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:35.691954 (+ 563us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":980452608}
I20260430 07:59:35.770784 12670 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:35.782167 12670 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.798249 12301 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:35.799414 12301 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.798264 12661 raft_consensus.cc:3055] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:35.802011 12661 raft_consensus.cc:740] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:35.803802 12661 consensus_queue.cc:260] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.807612 12779 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:35.828943 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.380508 12743 raft_consensus.cc:2749] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:35.848920 12717 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:35.853039 10532 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.854120 12721 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:35.855931 12890 log.cc:927] Time spent T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20260430 07:59:35.856779 12894 log.cc:927] Time spent T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.137s user 0.002s sys 0.000s
I20260430 07:59:35.708528 9934 rpcz_store.cc:276] 0430 07:59:33.450447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.450622 (+ 175us) service_pool.cc:225] Handling call
0430 07:59:33.461803 (+ 11181us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:33.557735 (+ 95932us) raft_consensus.cc:1534] Early marking committed up to index 3
0430 07:59:34.424209 (+866474us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b18003dc420. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb568e2a85c 00007fb5661c8c0b 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e2a95 00007fb56cc86837 00007fb56928dd87 00007fb56929d49f
0430 07:59:34.424371 (+ 162us) log.cc:844] Serialized 14 byte log entry
0430 07:59:34.496075 (+ 71704us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:34.496130 (+ 55us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:34.496174 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.699631 (+1203457us) spinlock_profiling.cc:243] Waited 95.5 ms on lock 0x7b540027f1d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:35.699743 (+ 112us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:35.700074 (+ 331us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18389,"prepare.queue_time_us":310511,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":8761,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":26514,"spinlock_wait_cycles":245938944,"thread_start_us":377480,"threads_started":2,"wal-append.queue_time_us":302515}
I20260430 07:59:35.862227 10568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54770 (request call id 405) took 1794 ms. Trace:
I20260430 07:59:35.862246 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1607) took 1781 ms. Trace:
I20260430 07:59:35.778378 12857 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:35.792727 12422 raft_consensus.cc:2749] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:35.862654 12661 raft_consensus.cc:3060] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:35.867527 9936 rpcz_store.cc:276] 0430 07:59:34.080999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.714687 (+1633688us) service_pool.cc:225] Handling call
0430 07:59:35.861340 (+146653us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:35.861857 (+ 517us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:35.861895 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:35.861964 (+ 69us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:35.861998 (+ 34us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.862059 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:35.862169 (+ 110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3072}
W20260430 07:59:35.873137 12905 log.cc:927] Time spent T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.189s user 0.002s sys 0.000s
I20260430 07:59:35.873450 9978 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:35.867059 10568 rpcz_store.cc:276] 0430 07:59:33.920378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.927844 (+ 7466us) service_pool.cc:225] Handling call
0430 07:59:33.928218 (+ 374us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:34.463120 (+534902us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.714265 (+1251145us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54004e0554. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:35.714744 (+ 479us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":623799168}
W20260430 07:59:35.875640 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:35.874912 10006 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1225) took 1085 ms. Trace:
I20260430 07:59:35.879118 12689 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:35.879536 9861 raft_consensus.cc:2393] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:35.880048 12301 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:35.880111 9978 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:35.880731 10005 consensus_meta.cc:287] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.541s user 0.014s sys 0.006s
I20260430 07:59:35.878626 10006 rpcz_store.cc:276] 0430 07:59:34.789833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:34.790027 (+ 194us) service_pool.cc:225] Handling call
0430 07:59:34.790506 (+ 479us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:35.214003 (+423497us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.874371 (+660368us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b540038d454. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:35.874692 (+ 321us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":971805952}
I20260430 07:59:35.893975 12858 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:35.900712 12410 consensus_meta.cc:287] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.526s user 0.010s sys 0.010s
I20260430 07:59:35.901227 12798 consensus_queue.cc:1059] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:35.902266 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:35.902505 12798 raft_consensus.cc:3055] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:35.903142 12798 raft_consensus.cc:740] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:35.903160 9854 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:35.897115 10005 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:35.919723 9859 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:35.893976 12737 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260430 07:59:35.904520 12904 log.cc:927] Time spent T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.167s user 0.000s sys 0.001s
I20260430 07:59:35.910485 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1163) took 2410 ms. Trace:
I20260430 07:59:35.926756 9932 raft_consensus.cc:1240] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260430 07:59:35.880002 12689 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.929560 9938 rpcz_store.cc:276] 0430 07:59:33.475992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.490556 (+ 14564us) service_pool.cc:225] Handling call
0430 07:59:35.876729 (+2386173us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b54001acf54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:35.881002 (+ 4273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":957182848}
I20260430 07:59:35.949959 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1850) took 1181 ms. Trace:
I20260430 07:59:35.910974 12784 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:35.926748 12410 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:36.000789 12784 raft_consensus.cc:493] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:35.904407 12798 consensus_queue.cc:260] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.004487 12784 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:36.005306 10008 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1853) took 2099 ms (client timeout 2197 ms). Trace:
W20260430 07:59:35.944981 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 89390.3us
I20260430 07:59:35.945796 12858 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.961755 12717 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:35.900786 12524 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:35.945508 9934 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 8. Current term is 9. Ops: []
I20260430 07:59:35.926683 12737 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:35.972054 9929 raft_consensus.cc:1240] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260430 07:59:35.971650 12714 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:35.995251 12799 consensus_queue.cc:237] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:35.996723 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 555) took 1249 ms. Trace:
I20260430 07:59:35.927668 12648 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:35.998301 12369 raft_consensus.cc:3055] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:36.018615 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 554) took 1272 ms. Trace:
I20260430 07:59:36.026180 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 556) took 1278 ms. Trace:
W20260430 07:59:36.003499 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 107778us
I20260430 07:59:35.966180 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1610) took 1831 ms. Trace:
I20260430 07:59:35.910058 12764 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:36.018486 9937 consensus_meta.cc:287] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.680s user 0.012s sys 0.010s
W20260430 07:59:36.005828 10008 rpcz_store.cc:269] 0430 07:59:33.905703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.905896 (+ 193us) service_pool.cc:225] Handling call
0430 07:59:36.004796 (+2098900us) spinlock_profiling.cc:243] Waited 813 ms on lock 0x7b540018e2d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:36.005070 (+ 274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1869581184}
I20260430 07:59:35.973152 9859 rpcz_store.cc:276] 0430 07:59:34.748055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.792014 (+1043959us) service_pool.cc:225] Handling call
0430 07:59:35.887008 (+ 94994us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:35.928803 (+ 41795us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.928971 (+ 168us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.031571 9858 rpcz_store.cc:276] 0430 07:59:34.746586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.684589 (+938003us) service_pool.cc:225] Handling call
0430 07:59:36.015111 (+330522us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8a3e 000000000035a3c9 00007fb56e1e08e0 00007fb56e2a9784 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:59:36.015308 (+ 197us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.018255 (+ 2947us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.018287 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.018349 (+ 62us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:36.018382 (+ 33us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.018436 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.018555 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2878080}
I20260430 07:59:36.032757 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1608) took 1941 ms. Trace:
I20260430 07:59:35.945561 9828 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:35.904819 12908 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:35.911528 12797 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:36.006829 12685 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:36.005775 12798 raft_consensus.cc:3060] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:36.016512 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1624) took 1877 ms. Trace:
I20260430 07:59:36.038620 9934 rpcz_store.cc:276] 0430 07:59:34.138967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.944420 (+1805453us) service_pool.cc:225] Handling call
0430 07:59:35.945039 (+ 619us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.016076 (+ 71037us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.016436 (+ 360us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.027396 9978 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:36.027459 9856 rpcz_store.cc:276] 0430 07:59:34.746891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.685902 (+939011us) service_pool.cc:225] Handling call
0430 07:59:35.995604 (+309702us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:59:35.995914 (+ 310us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:35.996275 (+ 361us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:35.996315 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:35.996374 (+ 59us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:35.996412 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.996491 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:35.996637 (+ 146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48584064}
I20260430 07:59:35.927882 12301 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:36.001217 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1590) took 1939 ms. Trace:
I20260430 07:59:36.021881 12830 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:36.031401 12369 raft_consensus.cc:740] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:36.007658 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1874) took 1215 ms. Trace:
I20260430 07:59:36.032063 9933 rpcz_store.cc:276] 0430 07:59:34.134555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.835326 (+1700771us) service_pool.cc:225] Handling call
0430 07:59:35.906707 (+ 71381us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:35.965695 (+ 58988us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:35.965728 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:35.965834 (+ 106us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:35.965866 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.965955 (+ 89us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:35.966090 (+ 135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.033258 9929 rpcz_store.cc:276] 0430 07:59:34.085835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.774660 (+1688825us) service_pool.cc:225] Handling call
0430 07:59:35.913619 (+138959us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:36.015691 (+102072us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.027364 (+ 11673us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.034837 9854 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:36.031859 9860 rpcz_store.cc:276] 0430 07:59:34.747174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.686964 (+939790us) service_pool.cc:225] Handling call
0430 07:59:36.025004 (+338040us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:59:36.025310 (+ 306us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.025651 (+ 341us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.025680 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.025749 (+ 69us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:36.025817 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.025880 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.026012 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45345664}
I20260430 07:59:35.946199 12788 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:36.022045 12664 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:35.961417 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1609) took 1827 ms. Trace:
I20260430 07:59:36.017877 12737 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:36.041313 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.047292 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1879) took 1028 ms. Trace:
I20260430 07:59:36.045046 9861 rpcz_store.cc:276] 0430 07:59:34.792778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.271484 (+478706us) service_pool.cc:225] Handling call
0430 07:59:35.974322 (+702838us) spinlock_profiling.cc:243] Waited 554 ms on lock 0x7b540029d454. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:36.007588 (+ 33266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1273415040}
I20260430 07:59:36.048003 9936 raft_consensus.cc:1275] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260430 07:59:36.048977 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1616) took 1912 ms. Trace:
I20260430 07:59:36.047073 9932 rpcz_store.cc:276] 0430 07:59:34.134179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.795678 (+1661499us) service_pool.cc:225] Handling call
0430 07:59:35.870217 (+ 74539us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:35.929838 (+ 59621us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:35.961343 (+ 31505us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25344}
I20260430 07:59:36.049404 9936 rpcz_store.cc:276] 0430 07:59:34.136406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.889359 (+1752953us) service_pool.cc:225] Handling call
0430 07:59:35.993780 (+104421us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.048723 (+ 54943us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.048927 (+ 204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.045436 9937 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:36.048441 9854 rpcz_store.cc:276] 0430 07:59:35.018900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.645078 (+626178us) service_pool.cc:225] Handling call
0430 07:59:35.902091 (+257013us) spinlock_profiling.cc:243] Waited 9.67 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8a3e 000000000035a3c9 00007fb56e1e08e0 00007fb56e2a9784 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:36.047217 (+145126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22251392}
I20260430 07:59:36.051468 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:36.042944 9931 rpcz_store.cc:276] 0430 07:59:34.062078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.598935 (+1536857us) service_pool.cc:225] Handling call
0430 07:59:35.701641 (+102706us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:35.710846 (+ 9205us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.000951 (+290105us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54001ee950. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:36.001129 (+ 178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":234050176}
I20260430 07:59:36.055634 9935 raft_consensus.cc:1240] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
W20260430 07:59:36.058228 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 64148.5us
I20260430 07:59:36.061529 12479 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:36.065809 12479 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:36.066892 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1853) took 1314 ms. Trace:
I20260430 07:59:36.067309 9858 rpcz_store.cc:276] 0430 07:59:34.748933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.044991 (+1296058us) service_pool.cc:225] Handling call
0430 07:59:36.062470 (+ 17479us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.062625 (+ 155us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.062639 (+ 14us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.062667 (+ 28us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:36.062684 (+ 17us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.062717 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.062802 (+ 85us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.064283 9938 raft_consensus.cc:1275] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260430 07:59:36.070914 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1626) took 1930 ms. Trace:
I20260430 07:59:36.071545 9938 rpcz_store.cc:276] 0430 07:59:34.139516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.039857 (+1900341us) service_pool.cc:225] Handling call
0430 07:59:36.061525 (+ 21668us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.068211 (+ 6686us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.069663 (+ 1452us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.066197 12479 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:36.075467 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 408) took 1684 ms. Trace:
I20260430 07:59:36.075830 9934 rpcz_store.cc:276] 0430 07:59:34.387800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.045360 (+1657560us) service_pool.cc:225] Handling call
0430 07:59:36.068632 (+ 23272us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.068761 (+ 129us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.068786 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.071517 (+ 2731us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:36.071557 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.071620 (+ 63us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.071750 (+ 130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.062376 12670 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:36.083293 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1598) took 2019 ms. Trace:
W20260430 07:59:36.087080 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 630985us
I20260430 07:59:36.059976 12685 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.086477 9935 rpcz_store.cc:276] 0430 07:59:34.064057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.599119 (+1535062us) service_pool.cc:225] Handling call
0430 07:59:36.055265 (+456146us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.056214 (+ 949us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.083097 (+ 26883us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b540036ced0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:36.083235 (+ 138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1037762304}
W20260430 07:59:36.093340 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.094417 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 411) took 1698 ms. Trace:
W20260430 07:59:36.094298 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 209400us
I20260430 07:59:36.094817 9929 rpcz_store.cc:276] 0430 07:59:34.396247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.059546 (+1663299us) service_pool.cc:225] Handling call
0430 07:59:36.081409 (+ 21863us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.090712 (+ 9303us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.090737 (+ 25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.090772 (+ 35us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:36.094145 (+ 3373us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.094224 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.094363 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.089505 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 409) took 1701 ms. Trace:
I20260430 07:59:36.096601 9933 rpcz_store.cc:276] 0430 07:59:34.388247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.052469 (+1664222us) service_pool.cc:225] Handling call
0430 07:59:36.082399 (+ 29930us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.087680 (+ 5281us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.087707 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.088745 (+ 1038us) raft_consensus.cc:1645] Marking committed up to 8
0430 07:59:36.088794 (+ 49us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.088867 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.089419 (+ 552us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.073603 12301 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.069408 12524 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:36.105441 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 785493us
I20260430 07:59:36.089020 12717 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:36.108574 9931 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 414) took 1649 ms. Trace:
I20260430 07:59:36.108909 9931 rpcz_store.cc:276] 0430 07:59:34.458992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.060433 (+1601441us) service_pool.cc:225] Handling call
0430 07:59:36.092554 (+ 32121us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.108162 (+ 15608us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.108202 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.108255 (+ 53us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:36.108294 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.108364 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.108515 (+ 151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.107007 12483 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:36.112592 9858 raft_consensus.cc:1240] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260430 07:59:36.113426 9859 raft_consensus.cc:1240] T cb149eebc54e462c951d45bca72b76e8 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 9. Current term is 10. Ops: []
I20260430 07:59:36.059412 12908 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:36.115594 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.116174 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1876) took 1266 ms. Trace:
I20260430 07:59:36.066406 12830 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:36.119016 10005 raft_consensus.cc:1240] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:36.089048 12369 consensus_queue.cc:260] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.120877 12830 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:36.117044 9863 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:36.115721 12256 consensus_queue.cc:1059] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.002s
I20260430 07:59:36.117625 12483 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:36.123080 12685 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:36.099259 12743 consensus_queue.cc:1059] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260430 07:59:36.085841 12664 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.124182 12483 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:36.118522 10008 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260430 07:59:36.124954 12301 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:36.125871 12664 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:36.127822 9856 raft_consensus.cc:1240] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:36.128290 9936 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260430 07:59:36.128749 9936 pending_rounds.cc:85] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 6
I20260430 07:59:36.121153 12369 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:36.129256 9936 pending_rounds.cc:107] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260430 07:59:36.129737 9936 raft_consensus.cc:2889] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:36.128669 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1856) took 1355 ms. Trace:
I20260430 07:59:36.130654 9856 rpcz_store.cc:276] 0430 07:59:34.772966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.104369 (+1331403us) service_pool.cc:225] Handling call
0430 07:59:36.127517 (+ 23148us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.128355 (+ 838us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.128590 (+ 235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.131877 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.132607 9856 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:36.133355 12788 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:36.133749 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.116712 12619 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:36.116640 9854 rpcz_store.cc:276] 0430 07:59:34.845675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.051806 (+1206131us) service_pool.cc:225] Handling call
0430 07:59:36.085683 (+ 33877us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.105130 (+ 19447us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.105159 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.105213 (+ 54us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:36.112108 (+ 6895us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.112179 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.112288 (+ 109us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:36.137458 12911 log.cc:927] Time spent T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260430 07:59:36.137995 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.138885 9828 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:36.139107 12773 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:36.134697 9856 raft_consensus.cc:2393] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 4: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:36.139936 12773 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.140673 9828 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:59:36.147248 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.116649 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1858) took 1068 ms. Trace:
I20260430 07:59:36.151336 9858 rpcz_store.cc:276] 0430 07:59:35.045090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.073962 (+1028872us) service_pool.cc:225] Handling call
0430 07:59:36.101702 (+ 27740us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:36.113060 (+ 11358us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.113220 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.116987 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1857) took 1229 ms. Trace:
I20260430 07:59:36.154629 9859 rpcz_store.cc:276] 0430 07:59:34.887537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.070026 (+1182489us) service_pool.cc:225] Handling call
0430 07:59:36.098059 (+ 28033us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.116698 (+ 18639us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.116935 (+ 237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.152734 9903 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:36.141155 9856 raft_consensus.cc:1275] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260430 07:59:36.156852 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1882) took 1079 ms. Trace:
I20260430 07:59:36.158998 9934 raft_consensus.cc:1240] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260430 07:59:36.158735 9856 rpcz_store.cc:276] 0430 07:59:35.077042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.140270 (+1063228us) service_pool.cc:225] Handling call
0430 07:59:36.140714 (+ 444us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.156638 (+ 15924us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.156810 (+ 172us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.141490 12670 raft_consensus.cc:2764] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260430 07:59:36.160717 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1181) took 1312 ms. Trace:
I20260430 07:59:36.160240 9937 raft_consensus.cc:1240] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 2. Current term is 3. Ops: []
I20260430 07:59:36.162959 9934 rpcz_store.cc:276] 0430 07:59:34.848059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.086087 (+1238028us) service_pool.cc:225] Handling call
0430 07:59:36.108359 (+ 22272us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:36.160307 (+ 51948us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.160655 (+ 348us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.163480 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 412) took 1705 ms. Trace:
I20260430 07:59:36.137255 12619 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:36.163983 9932 rpcz_store.cc:276] 0430 07:59:34.458249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.059785 (+1601536us) service_pool.cc:225] Handling call
0430 07:59:36.103123 (+ 43338us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.159735 (+ 56612us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.159751 (+ 16us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.163231 (+ 3480us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:36.163262 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.163332 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.163428 (+ 96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.163401 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 415) took 1673 ms. Trace:
I20260430 07:59:36.118901 12858 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:36.154265 12422 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:36.168524 9931 raft_consensus.cc:1240] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 9. Current term is 10. Ops: [7.5-9.6]
I20260430 07:59:36.165277 12483 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:36.169405 12483 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:36.162637 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:36.163806 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 416) took 1663 ms. Trace:
I20260430 07:59:36.167397 9937 rpcz_store.cc:276] 0430 07:59:34.489816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.073570 (+1583754us) service_pool.cc:225] Handling call
0430 07:59:36.110983 (+ 37413us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.163162 (+ 52179us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.163344 (+ 182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.172420 9938 rpcz_store.cc:276] 0430 07:59:34.500623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.074458 (+1573835us) service_pool.cc:225] Handling call
0430 07:59:36.103005 (+ 28547us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.159847 (+ 56842us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.159874 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.163567 (+ 3693us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:36.163613 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.163670 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.163759 (+ 89us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.179998 9978 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:36.181401 9855 raft_consensus.cc:1275] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260430 07:59:36.181854 9855 pending_rounds.cc:85] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 5
I20260430 07:59:36.182076 9855 pending_rounds.cc:107] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260430 07:59:36.182307 9855 raft_consensus.cc:2889] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:36.178004 12422 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:36.183686 12483 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:36.171069 12689 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:36.184401 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1185) took 1334 ms. Trace:
I20260430 07:59:36.184690 12865 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260430 07:59:36.185959 12773 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:36.184923 9933 rpcz_store.cc:276] 0430 07:59:34.850274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.106208 (+1255934us) service_pool.cc:225] Handling call
0430 07:59:36.164406 (+ 58198us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.184026 (+ 19620us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.184057 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.184123 (+ 66us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:36.184159 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.184220 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.184339 (+ 119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.183549 12664 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election won for term 9
I20260430 07:59:36.203243 12451 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:36.204048 12451 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.205834 12664 raft_consensus.cc:697] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:36.207252 12664 consensus_queue.cc:237] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:36.210525 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 50441us
I20260430 07:59:36.212342 12483 raft_consensus.cc:2804] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:36.216158 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:36.228525 12866 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:36.231459 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:36.232422 10567 raft_consensus.cc:2393] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 7: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:59:36.232719 12908 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:36.233845 12617 raft_consensus.cc:3055] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260430 07:59:36.234504 12617 raft_consensus.cc:740] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:36.213004 12483 raft_consensus.cc:697] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [term 7 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:36.236033 12483 consensus_queue.cc:237] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.235920 12617 consensus_queue.cc:260] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.247558 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:36.248164 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:36.249866 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1856) took 2447 ms. Trace:
I20260430 07:59:36.250478 9855 rpcz_store.cc:276] 0430 07:59:33.802197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.802354 (+ 157us) service_pool.cc:225] Handling call
0430 07:59:33.802695 (+ 341us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.182838 (+2380143us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.249412 (+ 66574us) spinlock_profiling.cc:243] Waited 2.36 s on lock 0x7b54001cd9d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:36.249749 (+ 337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5437980544}
I20260430 07:59:36.253626 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:36.255654 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1877) took 1409 ms. Trace:
I20260430 07:59:36.256227 12788 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260430 07:59:36.256167 9861 rpcz_store.cc:276] 0430 07:59:34.846148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.055469 (+1209321us) service_pool.cc:225] Handling call
0430 07:59:36.254585 (+199116us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e26b680 00007fb56e255fe9 00007fb56e255a92 00007fb5677bfd34
0430 07:59:36.254855 (+ 270us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.255224 (+ 369us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.255254 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.255338 (+ 84us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:36.255369 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.255448 (+ 79us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.255597 (+ 149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5871488}
I20260430 07:59:36.257995 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 413) took 1799 ms. Trace:
I20260430 07:59:36.258788 9936 rpcz_store.cc:276] 0430 07:59:34.458626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.058456 (+1599830us) service_pool.cc:225] Handling call
0430 07:59:36.086978 (+ 28522us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.130317 (+ 43339us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.257715 (+127398us) spinlock_profiling.cc:243] Waited 40.9 ms on lock 0x7b540027e2d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:36.257921 (+ 206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93982464}
W20260430 07:59:36.267112 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:36.269701 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 129228us
W20260430 07:59:36.271349 10534 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:36.273789 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 130327us
I20260430 07:59:36.276679 10568 raft_consensus.cc:2393] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 10: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:59:36.275446 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 559) took 1063 ms. Trace:
I20260430 07:59:36.199033 12619 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:36.277664 9859 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 558) took 1113 ms. Trace:
I20260430 07:59:36.280194 9859 rpcz_store.cc:276] 0430 07:59:35.164817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.159655 (+994838us) service_pool.cc:225] Handling call
0430 07:59:36.198912 (+ 39257us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.268405 (+ 69493us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.268423 (+ 18us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.268458 (+ 35us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:36.277328 (+ 8870us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.277425 (+ 97us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.277586 (+ 161us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.277895 9856 rpcz_store.cc:276] 0430 07:59:35.207720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.160521 (+952801us) service_pool.cc:225] Handling call
0430 07:59:36.197612 (+ 37091us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.266676 (+ 69064us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.266730 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.270903 (+ 4173us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:36.270930 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.270974 (+ 44us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.271067 (+ 93us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:36.284852 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 68804.8us
I20260430 07:59:36.268575 12524 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:36.286966 9855 raft_consensus.cc:1240] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
W20260430 07:59:36.289341 9977 connection.cc:254] RPC call timeout handler was delayed by -1.6078000310000107s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
W20260430 07:59:36.299374 9977 connection.cc:254] RPC call timeout handler was delayed by -1.524100030999989s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
I20260430 07:59:36.300005 9859 raft_consensus.cc:1275] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260430 07:59:36.300872 9856 raft_consensus.cc:1275] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260430 07:59:36.303211 9828 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:36.300364 9859 pending_rounds.cc:85] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 3
I20260430 07:59:36.304513 9859 pending_rounds.cc:107] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260430 07:59:34.904477 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260430 07:59:36.304751 9859 raft_consensus.cc:2889] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:36.294116 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:36.314419 9977 connection.cc:254] RPC call timeout handler was delayed by -1.4125000309999791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.188s for a response
W20260430 07:59:36.314920 9977 connection.cc:254] RPC call timeout handler was delayed by -1.3945000310000069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
W20260430 07:59:36.315215 9977 connection.cc:254] RPC call timeout handler was delayed by -1.3441000309999822s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
I20260430 07:59:36.306183 12617 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:36.319681 9856 raft_consensus.cc:1275] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260430 07:59:36.322578 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.324435 12749 raft_consensus.cc:2804] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:36.325425 12749 raft_consensus.cc:697] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:36.327056 12749 consensus_queue.cc:237] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.330075 12619 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:36.330907 9863 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1855) took 1566 ms (client timeout 1552 ms). Trace:
W20260430 07:59:36.331347 9977 connection.cc:254] RPC call timeout handler was delayed by -1.3000000309999677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2005s for a response
W20260430 07:59:36.332070 9977 connection.cc:254] RPC call timeout handler was delayed by -1.2748000309999838s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2033s for a response
W20260430 07:59:36.332491 9977 connection.cc:254] RPC call timeout handler was delayed by -1.1560000309999623s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21650000000000003s for a response
I20260430 07:59:36.336468 9903 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:36.332479 9863 rpcz_store.cc:269] 0430 07:59:34.764883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.703926 (+939043us) service_pool.cc:225] Handling call
0430 07:59:36.028739 (+324813us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8572 00007fb56f0651d9 00007fb56e1e05e0 00007fb56e2a976b 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:36.330532 (+301793us) spinlock_profiling.cc:243] Waited 69.3 ms on lock 0x7b540029e354. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:36.330730 (+ 198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166103168}
I20260430 07:59:36.328933 12479 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.362193 9858 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1883) took 1284 ms. Trace:
I20260430 07:59:36.366812 12479 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:36.367873 12479 consensus_queue.cc:1048] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:36.362665 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1194) took 1027 ms. Trace:
I20260430 07:59:36.379822 9930 rpcz_store.cc:276] 0430 07:59:35.280388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.308262 (+ 27874us) service_pool.cc:225] Handling call
0430 07:59:36.296534 (+988272us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b54002f08d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:36.307476 (+ 10942us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2229569792}
W20260430 07:59:36.389657 10532 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.363610 9858 rpcz_store.cc:276] 0430 07:59:35.077369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.152405 (+1075036us) service_pool.cc:225] Handling call
0430 07:59:36.180354 (+ 27949us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.202265 (+ 21911us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:36.266526 (+ 64261us) log.cc:844] Serialized 14 byte log entry
0430 07:59:36.361761 (+ 95235us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.361837 (+ 76us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:36.361929 (+ 92us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.361994 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.362134 (+ 140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":64051,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":1795,"raft.queue_time_us":3527,"thread_start_us":73338,"threads_started":1}
W20260430 07:59:36.424474 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 75264.2us
I20260430 07:59:36.424829 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:36.425952 10012 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 8.
I20260430 07:59:36.426298 12369 consensus_queue.cc:1059] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:36.427923 12369 raft_consensus.cc:3055] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:36.428649 12369 raft_consensus.cc:740] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:36.430629 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1881) took 1354 ms. Trace:
I20260430 07:59:36.431465 9854 rpcz_store.cc:276] 0430 07:59:35.076669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.139032 (+1062363us) service_pool.cc:225] Handling call
0430 07:59:36.139426 (+ 394us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.295852 (+156426us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.295883 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.295934 (+ 51us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:36.295973 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.430219 (+134246us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b540010ca54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:36.430290 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.430524 (+ 234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":355505920}
I20260430 07:59:36.437317 12422 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:36.437850 12912 log.cc:927] Time spent T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20260430 07:59:36.439801 9828 leader_election.cc:400] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7.
I20260430 07:59:36.441735 9935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1183) took 1592 ms. Trace:
I20260430 07:59:36.441999 12369 consensus_queue.cc:260] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.443593 12369 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:36.442361 9935 rpcz_store.cc:276] 0430 07:59:34.849193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.099091 (+1249898us) service_pool.cc:225] Handling call
0430 07:59:36.133494 (+ 34403us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:36.134424 (+ 930us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.134468 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:36.135436 (+ 968us) log.cc:844] Serialized 52 byte log entry
0430 07:59:36.161975 (+ 26539us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:36.162010 (+ 35us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.162055 (+ 45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:36.441390 (+279335us) raft_consensus.cc:1685] finished
0430 07:59:36.441423 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.441632 (+ 209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":149,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":137,"thread_start_us":25131,"threads_started":1,"wal-append.queue_time_us":124901}
I20260430 07:59:36.427815 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:36.539755 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.535960 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:36.477636 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 160033us
I20260430 07:59:36.535960 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:36.453164 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.472095 12451 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:36.457705 12915 log.cc:927] Time spent T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20260430 07:59:36.501592 9861 raft_consensus.cc:1240] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 3. Current term is 4. Ops: []
W20260430 07:59:36.526659 9977 connection.cc:254] RPC call timeout handler was delayed by -1.0021005689999356s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1965s for a response
W20260430 07:59:36.481546 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 75272.4us
W20260430 07:59:36.493031 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 84985.7us
I20260430 07:59:36.463591 9929 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1191) took 1563 ms. Trace:
I20260430 07:59:36.456677 12869 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:36.458425 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1875) took 1635 ms. Trace:
I20260430 07:59:36.453974 9828 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:36.474207 9857 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 557) took 1726 ms. Trace:
I20260430 07:59:36.454352 9933 raft_consensus.cc:1240] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 4. Current term is 5. Ops: []
I20260430 07:59:36.456259 12619 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:36.558035 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.483749 12478 raft_consensus.cc:3055] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:36.561914 9929 rpcz_store.cc:276] 0430 07:59:34.900436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.462026 (+1561590us) service_pool.cc:225] Handling call
0430 07:59:36.462618 (+ 592us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.463005 (+ 387us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.463044 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.463143 (+ 99us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:36.463219 (+ 76us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.463301 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.463463 (+ 162us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:36.500629 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.469744 12779 raft_consensus.cc:3055] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 8 LEADER]: Stepping down as leader of term 8
I20260430 07:59:36.456493 12914 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:36.579594 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 125762us
I20260430 07:59:36.474165 12618 consensus_queue.cc:1059] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.090s
I20260430 07:59:36.574409 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.569234 9857 rpcz_store.cc:276] 0430 07:59:34.747466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.795097 (+1047631us) service_pool.cc:225] Handling call
0430 07:59:35.875648 (+ 80551us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.039686 (+164038us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.039739 (+ 53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.043705 (+ 3966us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:36.046577 (+ 2872us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.473759 (+427182us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400206f54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:36.473844 (+ 85us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.474076 (+ 232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":341486464}
I20260430 07:59:36.575974 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:36.578212 9854 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:36.568583 9860 rpcz_store.cc:276] 0430 07:59:34.822716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.050446 (+1227730us) service_pool.cc:225] Handling call
0430 07:59:36.331569 (+281123us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.334503 (+ 2934us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:36.334551 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.334648 (+ 97us) raft_consensus.cc:1645] Marking committed up to 3
0430 07:59:36.334685 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.334755 (+ 70us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.458218 (+123463us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b540029e350. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:36.458359 (+ 141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":642363776}
I20260430 07:59:36.581313 12788 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:36.581724 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:36.582350 12664 consensus_queue.cc:1059] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:36.583024 12451 consensus_queue.cc:1059] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:36.475188 12799 consensus_queue.cc:1059] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.100s
I20260430 07:59:36.568724 12798 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:36.588482 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.589043 12749 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:36.476419 12859 consensus_queue.cc:1059] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.103s
I20260430 07:59:36.576679 12478 raft_consensus.cc:740] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:36.594195 12920 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:36.621129 12799 raft_consensus.cc:3055] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 9 LEADER]: Stepping down as leader of term 9
I20260430 07:59:36.464315 12737 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.602556 9863 raft_consensus.cc:3060] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:36.488723 12648 consensus_queue.cc:1048] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.090s
I20260430 07:59:36.491071 12865 consensus_queue.cc:1048] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
I20260430 07:59:36.614609 12798 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.616101 12664 raft_consensus.cc:3055] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Stepping down as leader of term 9
I20260430 07:59:36.580600 12779 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
W20260430 07:59:36.619800 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 43484.1us
I20260430 07:59:36.596513 12661 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:36.622128 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:36.620991 12920 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:36.619906 12670 raft_consensus.cc:3055] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:36.619336 12618 consensus_queue.cc:1243] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer 27f97a330f374849be58a168318c1ec6 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260430 07:59:36.621218 12478 consensus_queue.cc:260] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.627198 12749 raft_consensus.cc:697] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:36.624394 12799 raft_consensus.cc:740] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:36.631706 12648 consensus_queue.cc:1243] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer 27f97a330f374849be58a168318c1ec6 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260430 07:59:36.634004 12664 raft_consensus.cc:740] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:36.610477 12914 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:36.613624 12619 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.644930 12670 raft_consensus.cc:740] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:36.672065 12478 raft_consensus.cc:3060] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:36.655138 9757 catalog_manager.cc:5671] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260430 07:59:36.680756 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.687354 12869 raft_consensus.cc:3055] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:36.687742 12858 raft_consensus.cc:3055] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:36.688020 12869 raft_consensus.cc:740] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:36.688184 12858 raft_consensus.cc:740] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:36.689705 12869 consensus_queue.cc:260] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.692814 12869 raft_consensus.cc:3060] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:36.673879 12784 consensus_meta.cc:287] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.669s user 0.016s sys 0.000s
I20260430 07:59:36.694023 12799 consensus_queue.cc:260] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.694603 12784 raft_consensus.cc:515] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:36.695394 12799 raft_consensus.cc:3060] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Advancing to term 10
W20260430 07:59:36.702788 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:36.704559 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.711186 10006 raft_consensus.cc:2393] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 7: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:36.713712 12664 consensus_queue.cc:260] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:36.714861 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.722261 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:36.723198 10568 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:36.729893 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.738490 12619 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:36.742702 12914 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:36.715433 12664 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:36.731395 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.775185 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:36.766328 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:36.782620 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.784312 9857 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260430 07:59:36.793640 12670 consensus_queue.cc:260] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.794178 12695 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260430 07:59:36.795306 12670 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:36.805477 12779 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.806679 12779 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:36.689705 12858 consensus_queue.cc:260] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.816730 9861 raft_consensus.cc:1275] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260430 07:59:36.831501 10006 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:36.837059 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:36.840715 9759 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:48098 (request call id 192) took 23055 ms (client timeout 15000 ms). Trace:
I20260430 07:59:36.843601 9859 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 9
I20260430 07:59:36.853853 9978 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:36.855861 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1199) took 1573 ms. Trace:
I20260430 07:59:36.833045 12798 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:36.863831 9857 pending_rounds.cc:85] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 4
I20260430 07:59:36.864387 9857 pending_rounds.cc:107] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260430 07:59:36.868098 9857 raft_consensus.cc:2889] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:36.864342 9978 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:36.879549 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:36.882531 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:36.882717 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.842520 12914 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:36.897732 10006 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
W20260430 07:59:36.893919 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 335761us
I20260430 07:59:36.860369 9934 rpcz_store.cc:276] 0430 07:59:35.281997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.179927 (+897930us) service_pool.cc:225] Handling call
0430 07:59:36.234221 (+ 54294us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:36.234609 (+ 388us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:36.358562 (+123953us) log.cc:844] Serialized 14 byte log entry
0430 07:59:36.855512 (+496950us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:36.855566 (+ 54us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:36.855610 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:36.855674 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:36.855806 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2562,"prepare.queue_time_us":130119,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":33766,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"thread_start_us":598140,"threads_started":3,"wal-append.queue_time_us":298007}
I20260430 07:59:36.900616 12913 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:36.910614 12920 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:36.866154 12648 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:36.913729 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:36.903080 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 341074us
I20260430 07:59:36.924119 10006 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
W20260430 07:59:36.929454 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:36.930627 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:36.931070 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:36.900946 12737 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:36.908341 12743 consensus_queue.cc:1048] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.020s
I20260430 07:59:36.855803 12749 consensus_queue.cc:237] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:36.963361 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:36.965288 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:36.956848 9854 raft_consensus.cc:1275] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:36.971128 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:36.976431 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:36.978299 9903 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:36.988006 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:36.995502 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:36.843765 9759 rpcz_store.cc:269] 0430 07:59:13.785663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:13.785852 (+ 189us) service_pool.cc:225] Handling call
0430 07:59:36.840466 (+23054614us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0430 07:59:27.741938 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0430 07:59:27.865634 (+123696us) write_op.cc:432] Acquiring schema lock in shared mode
0430 07:59:27.865686 (+ 52us) write_op.cc:435] Acquired schema lock
0430 07:59:27.865708 (+ 22us) tablet.cc:662] Decoding operations
0430 07:59:28.105839 (+240131us) write_op.cc:620] Acquiring the partition lock for write op
0430 07:59:28.121836 (+ 15997us) write_op.cc:641] Partition lock acquired for write op
0430 07:59:28.127180 (+ 5344us) tablet.cc:685] Acquiring locks for 35 operations
0430 07:59:28.151744 (+ 24564us) tablet.cc:701] Row locks acquired
0430 07:59:28.151779 (+ 35us) write_op.cc:260] PREPARE: finished
0430 07:59:28.151977 (+ 198us) write_op.cc:270] Start()
0430 07:59:28.152134 (+ 157us) write_op.cc:276] Timestamp: P: 1777535968151900 usec, L: 0
0430 07:59:28.152169 (+ 35us) op_driver.cc:348] REPLICATION: starting
0430 07:59:28.153032 (+ 863us) log.cc:844] Serialized 13947 byte log entry
0430 07:59:28.723079 (+570047us) op_driver.cc:464] REPLICATION: finished
0430 07:59:28.815691 (+ 92612us) write_op.cc:301] APPLY: starting
0430 07:59:28.815811 (+ 120us) tablet.cc:1367] starting BulkCheckPresence
0430 07:59:28.816766 (+ 955us) tablet.cc:1370] finished BulkCheckPresence
0430 07:59:28.816941 (+ 175us) tablet.cc:1372] starting ApplyRowOperation cycle
0430 07:59:29.339120 (+522179us) tablet.cc:1383] finished ApplyRowOperation cycle
0430 07:59:29.493065 (+153945us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=35
0430 07:59:29.497820 (+ 4755us) write_op.cc:312] APPLY: finished
0430 07:59:29.498471 (+ 651us) log.cc:844] Serialized 230 byte log entry
0430 07:59:29.498710 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:59:29.498986 (+ 276us) write_op.cc:454] Released schema lock
0430 07:59:29.499077 (+ 91us) write_op.cc:341] FINISH: Updating metrics
0430 07:59:29.499649 (+ 572us) write_op.cc:489] Releasing partition, row and schema locks
0430 07:59:29.499693 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":46138,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":83673,"apply.run_cpu_time_us":14160,"apply.run_wall_time_us":683841,"num_ops":35,"prepare.queue_time_us":239723,"prepare.run_cpu_time_us":78725,"prepare.run_wall_time_us":667389,"raft.queue_time_us":77279,"raft.run_cpu_time_us":12994,"raft.run_wall_time_us":248898,"replication_time_us":499770,"thread_start_us":388849,"threads_started":4,"wal-append.queue_time_us":290025,"wal-append.run_cpu_time_us":7853,"wal-append.run_wall_time_us":2063579}]]}
I20260430 07:59:36.999164 9854 raft_consensus.cc:1275] T a48a2b75c3a34252bf57a4f634457236 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:37.001608 12921 consensus_queue.cc:1048] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:37.002185 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:37.002954 10568 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:36.934641 12914 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:37.014663 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 300103us
I20260430 07:59:37.021572 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:37.015018 9859 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:37.026582 9978 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:37.030788 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 87236.9us
I20260430 07:59:37.031534 12929 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:37.031772 9903 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:37.032220 12929 raft_consensus.cc:697] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [term 8 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:37.035419 12478 raft_consensus.cc:2804] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election won for term 9
W20260430 07:59:37.040982 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.044060 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:37.046814 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.050078 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:37.051591 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:37.052834 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:37.053201 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 294181us
I20260430 07:59:37.061929 9937 raft_consensus.cc:1240] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:37.062312 9938 raft_consensus.cc:1275] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260430 07:59:37.062762 9938 pending_rounds.cc:85] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 3
I20260430 07:59:37.063063 9938 pending_rounds.cc:107] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260430 07:59:37.050590 12798 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:37.063452 9938 raft_consensus.cc:2889] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:37.063632 12798 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.065379 10568 raft_consensus.cc:2393] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 8: Already voted for candidate d66db415f0f0402fa431349afbf59888 in this term.
I20260430 07:59:37.066311 12798 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:37.049676 12929 consensus_queue.cc:237] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.067070 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:37.045953 12869 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260430 07:59:37.076418 12923 log.cc:927] Time spent T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.277s user 0.002s sys 0.000s
I20260430 07:59:37.074961 9757 catalog_manager.cc:5671] T 02a63423ae464eb7b415a38419524150 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 4 to 7, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 7 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:59:37.035138 12926 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.055s
I20260430 07:59:37.081810 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.083554 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.084301 9932 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:37.082005 12914 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:37.088294 9930 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:37.085181 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.093878 9929 raft_consensus.cc:1275] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260430 07:59:37.098055 9932 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:37.046290 12784 leader_election.cc:290] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:37.097374 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 456184us
I20260430 07:59:37.104439 10533 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:37.102738 9828 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:37.117722 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.118973 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.119992 9935 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:37.121308 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.106098 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.084939 12721 consensus_queue.cc:1048] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260430 07:59:37.057583 12478 raft_consensus.cc:697] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [term 9 LEADER]: Becoming Leader. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:37.132622 9936 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 10.
I20260430 07:59:37.133558 9937 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:37.134387 9938 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 10.
I20260430 07:59:37.141475 9931 raft_consensus.cc:1275] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260430 07:59:37.142153 9903 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:37.164434 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:37.167268 9863 consensus_meta.cc:287] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.531s user 0.014s sys 0.001s
I20260430 07:59:37.168862 9863 raft_consensus.cc:2468] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 7.
I20260430 07:59:37.160959 9862 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:37.161296 12619 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260430 07:59:37.194002 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 123857us
I20260430 07:59:37.106722 12869 consensus_queue.cc:1059] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.069s
W20260430 07:59:37.197095 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.205403 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.206544 10006 raft_consensus.cc:2393] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 8: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
W20260430 07:59:37.208488 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.167940 12926 raft_consensus.cc:2749] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:37.202103 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e51196f9c4fb44bf8f6296f5394417bb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.190665 12930 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260430 07:59:37.230011 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 106367us
I20260430 07:59:37.248749 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:37.249759 10008 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:37.264215 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.266140 10004 raft_consensus.cc:1275] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260430 07:59:37.266158 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.267367 10003 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:37.269462 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.270087 9929 raft_consensus.cc:1275] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260430 07:59:37.270844 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.271143 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:37.271514 9935 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:37.272397 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.272404 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.273253 9929 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:37.273502 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.274075 9938 raft_consensus.cc:1275] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260430 07:59:37.274325 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.278945 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:37.279693 9929 raft_consensus.cc:3060] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:37.279937 9977 leader_election.cc:336] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.048s (SENT)
I20260430 07:59:37.280511 9977 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:37.284497 10012 raft_consensus.cc:1275] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:37.232553 12930 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:37.285758 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1851) took 2537 ms. Trace:
I20260430 07:59:37.285976 12930 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:37.222168 12799 raft_consensus.cc:3055] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 9 LEADER]: Stepping down as leader of term 9
I20260430 07:59:37.286399 9862 rpcz_store.cc:276] 0430 07:59:34.748344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:35.778977 (+1030633us) service_pool.cc:225] Handling call
0430 07:59:35.877124 (+ 98147us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:37.176188 (+1299064us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:37.285445 (+109257us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540029cf54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:37.285682 (+ 237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2929618688}
I20260430 07:59:37.287417 9938 pending_rounds.cc:85] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 4
I20260430 07:59:37.293849 9938 pending_rounds.cc:107] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260430 07:59:37.294216 9938 raft_consensus.cc:2889] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:37.275770 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:37.295142 9937 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:37.276180 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.274556 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1caba957d05d44788146efd70334d6fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:37.296157 9935 raft_consensus.cc:3060] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:37.301240 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.302032 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.302937 9938 raft_consensus.cc:2410] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260430 07:59:37.305783 12784 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:37.306346 9937 raft_consensus.cc:1275] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260430 07:59:37.276551 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:37.310463 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 86484.6us
W20260430 07:59:37.315073 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:37.323019 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:37.317965 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 175351us
I20260430 07:59:37.315214 10008 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
W20260430 07:59:37.327045 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 114318us
I20260430 07:59:37.339351 9929 raft_consensus.cc:2468] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
W20260430 07:59:37.343043 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.345228 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.307973 12799 raft_consensus.cc:740] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:37.349344 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:37.339532 12661 consensus_queue.cc:1059] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20260430 07:59:37.347647 12619 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:37.375897 12942 consensus_queue.cc:1059] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:37.385252 12942 raft_consensus.cc:3055] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Stepping down as leader of term 2
I20260430 07:59:37.385824 12942 raft_consensus.cc:740] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:37.386996 12942 consensus_queue.cc:260] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:37.388062 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:37.388803 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.260077 12478 consensus_queue.cc:237] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.348968 12788 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:37.407021 12788 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:37.410665 12788 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:37.370863 9977 leader_election.cc:336] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.171s (SENT)
I20260430 07:59:37.309969 9757 catalog_manager.cc:5671] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:59:37.411732 9977 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:37.413314 12737 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:37.413341 9757 catalog_manager.cc:5671] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260430 07:59:37.418100 9757 catalog_manager.cc:5671] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 reported cstate change: term changed from 1 to 5, leader changed from 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) to d66db415f0f0402fa431349afbf59888 (127.8.119.132). New cstate: current_term: 5 leader_uuid: "d66db415f0f0402fa431349afbf59888" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260430 07:59:37.420637 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 71460.2us
W20260430 07:59:37.419761 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 62446.1us
I20260430 07:59:37.422011 9828 leader_election.cc:304] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:37.422559 9977 leader_election.cc:336] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.278s (SENT)
I20260430 07:59:37.423231 9977 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:37.424088 12737 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:37.424587 12866 raft_consensus.cc:2764] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260430 07:59:37.430153 9935 raft_consensus.cc:2468] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:37.367894 12784 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:37.449900 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.389153 12942 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 2 FOLLOWER]: Advancing to term 3
W20260430 07:59:37.458173 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 262095us
I20260430 07:59:37.447024 9828 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:37.453037 12661 consensus_queue.cc:1243] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer 27f97a330f374849be58a168318c1ec6 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260430 07:59:37.470345 10008 raft_consensus.cc:1275] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260430 07:59:37.471128 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:37.469017 12933 log.cc:927] Time spent T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20260430 07:59:37.472828 12451 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:37.473930 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:37.474691 10005 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:37.476814 12931 consensus_queue.cc:1059] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260430 07:59:37.478072 12931 raft_consensus.cc:3055] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Stepping down as leader of term 4
I20260430 07:59:37.478456 12931 raft_consensus.cc:740] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:37.478624 12812 consensus_queue.cc:1059] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.040s
I20260430 07:59:37.479149 12869 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:37.479326 12931 consensus_queue.cc:260] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:37.479712 12869 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:37.482767 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.474563 12945 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:37.479645 12812 raft_consensus.cc:3055] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:37.488639 10534 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:37.435323 12619 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.487843 10533 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:37.501737 9828 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:37.501858 12451 raft_consensus.cc:697] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:37.515260 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.513857 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.516201 10006 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:37.482228 12931 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:37.523715 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.457531 12784 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.537052 12784 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:37.538429 10004 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:37.512813 12914 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:37.539047 9828 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:37.539090 12914 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.518998 12945 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.524591 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.542717 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:37.543316 9855 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:37.538148 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:37.532874 12944 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:37.536935 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.542359 10007 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:37.542925 10008 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:37.536901 10009 raft_consensus.cc:1240] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [term 10 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 9. Current term is 10. Ops: []
I20260430 07:59:37.466656 12908 consensus_queue.cc:1048] T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260430 07:59:37.496156 12812 raft_consensus.cc:740] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:37.541409 12914 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:37.539204 12749 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:37.567389 12749 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:37.568135 12930 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:37.569684 9978 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:37.570186 12749 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:37.570276 12619 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:37.465376 12799 consensus_queue.cc:260] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:37.543785 12721 raft_consensus.cc:2804] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:37.583040 10008 raft_consensus.cc:1240] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
W20260430 07:59:37.587353 12949 log.cc:927] Time spent T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260430 07:59:37.589615 12921 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:37.589828 12945 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:37.590241 12921 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.591141 12930 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:37.583439 12749 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:37.592015 12749 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:37.592304 12921 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:37.592402 12749 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:37.602418 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.578105 9930 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:37.604127 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:37.614384 10005 raft_consensus.cc:2468] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
W20260430 07:59:37.627987 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 140185us
I20260430 07:59:37.538046 12451 consensus_queue.cc:237] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.583020 12955 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:37.609462 12784 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260430 07:59:37.644541 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.615406 12721 raft_consensus.cc:697] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [term 7 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:37.662654 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:37.651547 10533 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:37.676273 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.677191 9930 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:37.673154 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.684289 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.686326 10007 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:37.687992 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:37.696254 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:37.696934 10005 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:37.697607 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:37.645556 12784 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260430 07:59:37.701226 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 160274us
I20260430 07:59:37.659442 12869 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:37.696660 12799 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:37.708032 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:37.709404 12619 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:37.718672 9861 raft_consensus.cc:1275] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:37.671411 12955 raft_consensus.cc:697] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:37.725569 10003 raft_consensus.cc:1275] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:37.674602 12830 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:37.699662 12784 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:37.702987 12617 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:37.744922 10005 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 9.
I20260430 07:59:37.746505 9828 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:37.730444 12858 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:37.738509 12664 consensus_meta.cc:287] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.971s user 0.013s sys 0.001s
I20260430 07:59:37.731113 12963 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:37.737838 12869 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:37.756574 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 61537.6us
I20260430 07:59:37.664292 12812 consensus_queue.cc:260] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.765789 12955 consensus_queue.cc:237] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.766467 12812 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:37.768944 12927 raft_consensus.cc:3055] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:37.754340 12959 consensus_queue.cc:1048] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.016s
I20260430 07:59:37.764464 12749 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:37.795639 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.799551 9828 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:37.799662 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.798091 12927 raft_consensus.cc:740] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:37.755589 12963 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:37.800263 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e537f351c19b43fd8b0b0a89cf897487" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:37.798573 12860 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:37.803835 12956 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:37.809175 9828 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:37.750258 12617 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:37.810777 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 113034us
I20260430 07:59:37.817816 12860 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.807941 12963 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:37.769412 12721 consensus_queue.cc:237] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:37.826566 12749 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:37.826598 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:37.828807 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:37.829471 12927 consensus_queue.cc:260] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:37.831291 12951 log.cc:927] Time spent T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.357s user 0.002s sys 0.000s
I20260430 07:59:37.804018 12664 consensus_queue.cc:1048] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260430 07:59:37.886976 12617 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:37.887323 12779 raft_consensus.cc:493] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:37.887847 12779 raft_consensus.cc:515] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:37.837599 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.836964 12830 raft_consensus.cc:2955] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:37.897957 12927 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:37.893407 9857 raft_consensus.cc:2955] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
W20260430 07:59:37.904685 12965 log.cc:927] Time spent T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.272s user 0.000s sys 0.004s
W20260430 07:59:37.908176 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:37.917738 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.922923 12779 leader_election.cc:290] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
W20260430 07:59:37.838928 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.808501 12962 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:37.843756 9828 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:37.798424 12869 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.804418 12926 consensus_queue.cc:1048] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
W20260430 07:59:37.856956 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.882102 12956 raft_consensus.cc:697] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:37.855135 12619 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:37.929241 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:37.935778 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 92102.6us
I20260430 07:59:37.932984 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:37.925637 12962 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:37.978399 12962 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:37.980334 9931 raft_consensus.cc:1275] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:37.980710 12945 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:37.981591 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:37.981967 12969 log.cc:927] Time spent T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260430 07:59:37.975026 12478 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:37.981670 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.982692 12478 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:37.982372 9938 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:37.984881 12478 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:37.986356 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.986212 12963 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:37.986992 9935 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
I20260430 07:59:37.988268 12963 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:37.988943 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:37.989559 9929 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
I20260430 07:59:37.992813 10003 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:37.993149 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.993666 10003 raft_consensus.cc:606] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:37.994028 10003 raft_consensus.cc:614] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:37.995421 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 11 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.997063 10010 raft_consensus.cc:1240] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
W20260430 07:59:37.997390 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 479008us
I20260430 07:59:37.998176 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:37.998840 10012 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:37.999742 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.002310 12860 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:38.003032 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:38.003177 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:38.004601 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:38.005970 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:38.006110 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:38.006634 10005 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:38.007324 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:38.010011 9933 raft_consensus.cc:1275] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260430 07:59:38.010233 10006 raft_consensus.cc:1275] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260430 07:59:38.010640 10006 pending_rounds.cc:85] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032: Aborting all ops after (but not including) 4
I20260430 07:59:38.010888 10006 pending_rounds.cc:107] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260430 07:59:38.011200 10006 raft_consensus.cc:2889] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:38.011132 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:38.011904 12869 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:37.981401 12945 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:38.013684 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.930559 12779 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:38.033175 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.036880 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.006686 10566 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 6.
I20260430 07:59:38.038343 10005 raft_consensus.cc:2410] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260430 07:59:37.935695 12619 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:38.041164 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:37.938062 12670 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:38.048092 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:38.049216 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.050839 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.052336 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:38.052512 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.052670 10004 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 3.
W20260430 07:59:38.052690 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 122517us
I20260430 07:59:37.974992 12749 consensus_queue.cc:1048] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20260430 07:59:38.058528 12860 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:38.048614 12670 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:38.059026 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.059767 12956 consensus_queue.cc:237] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.061342 10012 raft_consensus.cc:1275] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:38.048132 12619 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:38.052642 12931 consensus_meta.cc:287] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.507s user 0.007s sys 0.005s
W20260430 07:59:38.063947 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:38.058652 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.066457 10533 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:38.058974 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.073650 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:38.071820 9903 connection.cc:254] RPC call timeout handler was delayed by -1.9070929760000013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17210000000000003s for a response
W20260430 07:59:38.126333 9903 connection.cc:254] RPC call timeout handler was delayed by -1.2296630689999688s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18510000000000001s for a response
I20260430 07:59:38.074913 10006 raft_consensus.cc:1275] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260430 07:59:38.067937 9828 leader_election.cc:304] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:38.070636 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:38.058789 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de66bf2ed62e49aa95b1deff0383d037" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
W20260430 07:59:38.127990 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 61700.4us
I20260430 07:59:38.076979 12670 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:38.051347 12779 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.132174 10533 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:59:38.146225 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 78336us
I20260430 07:59:38.147442 9828 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:59:38.226001 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.122256 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:38.228020 9828 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:38.073211 12869 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:38.151218 10004 raft_consensus.cc:2393] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
W20260430 07:59:38.234504 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 102416us
I20260430 07:59:38.238400 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:38.124722 12963 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:38.245187 9903 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:38.144716 12908 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:38.115674 12860 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:38.251787 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.225904 12451 raft_consensus.cc:2749] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:38.131022 12661 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:59:38.252890 9977 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.238433 12812 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:38.254518 12664 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:38.141680 12784 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.269449 12947 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 9 FOLLOWER]: Leader election won for term 9
I20260430 07:59:38.270712 10005 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:38.276151 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:38.273567 12980 log.cc:927] Time spent T 00000000000000000000000000000000 P b5dc045545db48eba3c35052f59c6ced: Append to log took a long time: real 0.129s user 0.000s sys 0.002s
I20260430 07:59:38.277621 10012 raft_consensus.cc:1240] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260430 07:59:38.282632 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.249651 12908 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:38.257807 12926 raft_consensus.cc:2749] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260430 07:59:38.289911 10007 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:38.291334 9828 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:38.292798 12979 log.cc:927] Time spent T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
W20260430 07:59:38.298475 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.300014 9858 raft_consensus.cc:1240] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
W20260430 07:59:38.302860 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 67899.6us
I20260430 07:59:38.252125 12963 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.279278 12812 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:38.289455 12908 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:38.296180 12648 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:38.258797 12869 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.295704 12743 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:38.295964 12971 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:38.306721 12664 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:38.306881 12978 log.cc:927] Time spent T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260430 07:59:38.323411 9934 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:38.278714 12661 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.297614 12947 raft_consensus.cc:697] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [term 9 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:38.333669 9859 raft_consensus.cc:1240] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:38.338433 12984 consensus_queue.cc:1059] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260430 07:59:38.339803 12981 raft_consensus.cc:3055] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:38.340210 12981 raft_consensus.cc:740] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:38.341042 12981 consensus_queue.cc:260] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:38.342422 12981 raft_consensus.cc:3060] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Advancing to term 7
W20260430 07:59:38.347440 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.347121 12947 consensus_queue.cc:237] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:38.348654 12779 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:38.349246 12648 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.351451 12648 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:38.353331 12661 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:38.319293 12945 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:38.328588 12743 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.357448 10007 raft_consensus.cc:1275] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:38.359596 12963 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:38.308521 12812 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:38.366628 12784 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:38.308656 12617 consensus_queue.cc:1059] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260430 07:59:38.375137 12617 raft_consensus.cc:3055] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:38.375685 12617 raft_consensus.cc:740] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:38.378719 12869 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:38.381563 9903 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:38.383248 12860 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:38.384334 9903 leader_election.cc:304] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:38.385504 12945 raft_consensus.cc:2749] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:38.385995 10007 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: [6.4-7.5]
I20260430 07:59:38.384295 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.327032 12971 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:38.389860 9976 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.390388 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.391875 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.385743 9903 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:38.394691 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:38.402333 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 87761.3us
I20260430 07:59:38.403770 9903 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:38.405201 9903 leader_election.cc:304] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:38.419423 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.417016 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.422839 12945 raft_consensus.cc:2749] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:38.435277 12963 consensus_queue.cc:1048] T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260430 07:59:38.439622 9828 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:59:38.400460 12830 consensus_meta.cc:287] T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.501s user 0.013s sys 0.001s
I20260430 07:59:38.384282 12860 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260430 07:59:38.482208 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.449491 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:38.453369 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:38.455307 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.376840 12617 consensus_queue.cc:260] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.487169 9856 raft_consensus.cc:1275] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260430 07:59:38.487504 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.456861 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.488113 9938 raft_consensus.cc:2468] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:38.420055 12661 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:38.445808 12451 consensus_queue.cc:1059] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260430 07:59:38.537452 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.460258 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:38.405563 12743 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:38.415958 12869 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:38.463045 12971 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:38.465644 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.465510 12962 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:38.466373 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.467031 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.469709 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.471969 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.475060 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:38.579511 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 273744us
I20260430 07:59:38.466876 12664 raft_consensus.cc:2749] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:38.480073 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.584249 10004 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:38.485965 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.585253 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:38.486505 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:38.488902 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:38.489969 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:38.492015 9826 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.484133 12963 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:38.516505 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:38.596095 9933 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:38.596443 9936 raft_consensus.cc:2368] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 8. Current term is 9.
I20260430 07:59:38.586120 9938 raft_consensus.cc:2376] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9. Re-sending same reply.
I20260430 07:59:38.518000 9935 raft_consensus.cc:1275] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260430 07:59:38.540578 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.514690 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:38.579843 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.587013 9930 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:38.456115 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:38.618337 9929 raft_consensus.cc:2393] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 7: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:38.586432 9931 raft_consensus.cc:2376] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9. Re-sending same reply.
I20260430 07:59:38.620220 10003 raft_consensus.cc:2410] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260430 07:59:38.478952 9828 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:38.574760 12869 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:38.576115 12860 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:38.486166 12955 consensus_queue.cc:1048] T bddad6955cd64efd850fddc61341be7e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.137s
I20260430 07:59:38.583705 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:38.661398 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.668804 9932 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 10.
I20260430 07:59:38.618937 12963 raft_consensus.cc:493] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:38.595662 12779 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
W20260430 07:59:38.678665 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 199767us
W20260430 07:59:38.682499 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.661410 12869 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:38.691123 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:59:38.686137 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 110107us
I20260430 07:59:38.704295 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.627946 12670 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.710189 9932 raft_consensus.cc:2393] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 5: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:38.667629 12945 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:38.714387 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.685783 12664 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:38.735741 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.740396 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:38.742159 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 51179.5us
I20260430 07:59:38.678231 12963 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:38.748476 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:38.749840 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.668828 12860 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.751504 9863 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:38.758307 9936 raft_consensus.cc:2376] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 10. Re-sending same reply.
I20260430 07:59:38.758219 9828 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:38.758530 10009 raft_consensus.cc:2955] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 3 to 6, VOTER 77e19ab653f540339c1c613fad1451b5 (127.8.119.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } } }
I20260430 07:59:38.753386 12927 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:38.768653 9858 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260430 07:59:38.764338 12927 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:38.775074 10531 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.739400 12959 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260430 07:59:38.714264 12956 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.053s
I20260430 07:59:38.754572 12664 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
W20260430 07:59:38.809417 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 66303.3us
I20260430 07:59:38.809092 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.816671 9937 raft_consensus.cc:2393] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:38.735636 12992 consensus_queue.cc:1048] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.048s
W20260430 07:59:38.823448 12997 log.cc:927] Time spent T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260430 07:59:38.824699 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.825412 9936 raft_consensus.cc:2376] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7. Re-sending same reply.
W20260430 07:59:38.826011 12988 log.cc:927] Time spent T 7c393abc9c604fc9aff3cedf85f68725 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20260430 07:59:38.826423 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:38.827303 12989 log.cc:927] Time spent T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
W20260430 07:59:38.837246 12991 log.cc:927] Time spent T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
W20260430 07:59:38.845063 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.772642 12985 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:38.782143 12955 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260430 07:59:38.781679 12959 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:38.801383 12962 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:38.866562 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.876950 9929 raft_consensus.cc:2393] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:38.803577 12661 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:38.821223 12664 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:38.883157 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 125047us
I20260430 07:59:38.887984 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:38.889880 9977 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:38.793677 12960 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20260430 07:59:38.757697 12945 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.851141 12670 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:38.859014 12955 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:38.875237 12959 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:38.897521 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:38.895253 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.905646 9936 raft_consensus.cc:2376] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 10. Re-sending same reply.
I20260430 07:59:38.905871 9929 raft_consensus.cc:2393] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 10: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:38.907090 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.873353 12830 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:48088 (request call id 9) took 5079 ms. Trace:
I20260430 07:59:38.909098 9936 raft_consensus.cc:2410] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260430 07:59:38.861909 12784 consensus_queue.cc:1048] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260430 07:59:38.911494 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:38.911929 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.912292 9929 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
W20260430 07:59:38.916525 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.929836 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:38.931044 9937 raft_consensus.cc:2410] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260430 07:59:38.893261 12866 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260430 07:59:38.932183 12866 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:38.932710 12866 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:38.940212 9903 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:38.944872 9932 raft_consensus.cc:1275] T 1caba957d05d44788146efd70334d6fe P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Refusing update from remote peer 48e81934d52a4e93be5171ceea193032: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260430 07:59:38.904590 12955 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:38.950017 9977 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:38.904820 12992 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260430 07:59:38.955092 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:38.888402 12661 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:38.870653 12985 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:38.974108 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.976629 9855 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:38.983309 9904 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:38.986732 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:38.945931 12779 consensus_queue.cc:1048] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.036s
I20260430 07:59:38.936255 12670 consensus_queue.cc:1048] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.037s
I20260430 07:59:38.980638 12992 raft_consensus.cc:2749] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260430 07:59:39.002071 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 82476.8us
W20260430 07:59:39.011536 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 71449.3us
I20260430 07:59:39.015913 10006 raft_consensus.cc:1275] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260430 07:59:38.999271 12619 consensus_meta.cc:287] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.854s user 0.009s sys 0.004s
I20260430 07:59:39.017748 9903 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:59:39.053424 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 72321.4us
I20260430 07:59:39.061475 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:39.010443 12927 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.013053 12860 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.015899 12960 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:39.073879 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 56201.3us
W20260430 07:59:39.061784 12812 consensus_meta.cc:287] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.696s user 0.011s sys 0.003s
I20260430 07:59:39.060081 12619 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.099949 9903 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:39.103899 12985 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:39.088135 12927 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:39.109741 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:39.076412 13018 raft_consensus.cc:2749] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:39.069764 12984 consensus_queue.cc:1048] T de66bf2ed62e49aa95b1deff0383d037 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
W20260430 07:59:39.123416 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 212339us
I20260430 07:59:39.095049 12945 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:39.126454 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:39.128110 9861 raft_consensus.cc:1240] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:39.097707 13008 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:39.142716 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 251651us
W20260430 07:59:39.143847 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 47790.2us
I20260430 07:59:39.145015 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:39.145953 9828 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:39.088840 12812 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:39.150141 9903 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:39.148564 12962 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 07:59:39.168213 12962 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:39.168380 12986 raft_consensus.cc:3055] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:39.168642 12962 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:39.168946 12986 raft_consensus.cc:740] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:39.170042 12986 consensus_queue.cc:260] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.167537 9859 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 2.
I20260430 07:59:39.174674 10560 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b"
dest_uuid: "d66db415f0f0402fa431349afbf59888"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:39574
I20260430 07:59:39.175403 10560 raft_consensus.cc:606] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:39.175837 12869 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:39.170073 9828 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:39.122429 12927 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:59:39.176946 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 52573.6us
I20260430 07:59:39.179096 10534 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:59:39.176193 9934 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1200) took 2227 ms (client timeout 1780 ms). Trace:
I20260430 07:59:39.153196 9903 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
W20260430 07:59:39.183120 9934 rpcz_store.cc:269] 0430 07:59:36.948507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:36.948680 (+ 173us) service_pool.cc:225] Handling call
0430 07:59:39.176041 (+2227361us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54003241d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:39.176131 (+ 90us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2405481216}
W20260430 07:59:39.180164 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:39.181245 9858 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:39.185564 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:39.087424 12960 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:39.125991 12860 raft_consensus.cc:2749] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:39.191455 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:39.199146 10006 raft_consensus.cc:2368] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9.
I20260430 07:59:39.199391 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:39.193967 9977 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:39.200953 9934 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260430 07:59:39.201457 9854 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:39.200026 10008 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
W20260430 07:59:39.197074 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 43994.5us
I20260430 07:59:39.202692 9903 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:39.201210 9977 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:39.204442 9933 raft_consensus.cc:2393] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 6: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:39.205917 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:39.208055 10010 raft_consensus.cc:1275] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260430 07:59:39.192058 12927 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:39.210309 12648 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:39.229988 12648 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:39.149785 13008 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:39.230598 12648 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:39.211273 9932 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260430 07:59:39.232353 9932 pending_rounds.cc:85] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Aborting all ops after (but not including) 4
I20260430 07:59:39.232648 9932 pending_rounds.cc:107] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260430 07:59:39.232946 9932 raft_consensus.cc:2889] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:39.210183 10006 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 in current term 8: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:39.234081 12960 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.180459 13018 raft_consensus.cc:2749] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 07:59:39.192433 12986 raft_consensus.cc:3060] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:39.234910 12869 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:39.238176 10009 raft_consensus.cc:1275] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260430 07:59:39.239950 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:39.219036 9937 raft_consensus.cc:1275] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:39.242802 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:39.243398 9937 raft_consensus.cc:3060] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
W20260430 07:59:39.243566 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:39.244843 12987 raft_consensus.cc:2804] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260430 07:59:39.245375 12987 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:39.245738 12987 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:39.218914 9903 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:39.246361 13014 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:39.247535 13014 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:39.250885 10008 raft_consensus.cc:1275] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260430 07:59:39.261974 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:39.262701 9931 raft_consensus.cc:2468] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:39.263597 12927 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.250874 13014 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.223482 12962 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.268049 12962 leader_election.cc:290] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.232509 13008 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:39.234127 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:39.269824 9932 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 3.
I20260430 07:59:39.257058 9937 raft_consensus.cc:2468] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:39.265745 12927 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:39.295162 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 61576.1us
I20260430 07:59:39.194667 12451 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:39.191659 12661 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.220305 13012 raft_consensus.cc:2804] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:39.278687 10012 raft_consensus.cc:1275] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:39.249627 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:39.216105 10004 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
W20260430 07:59:39.220017 13013 log.cc:927] Time spent T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260430 07:59:39.200380 12799 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:39.310405 9936 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:39.314211 9977 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:39.310740 10004 raft_consensus.cc:606] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:39.316023 10004 raft_consensus.cc:614] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:39.315402 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:39.317265 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 73016.9us
I20260430 07:59:38.913816 12830 rpcz_store.cc:276] 0430 07:59:33.770183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:33.770365 (+ 182us) service_pool.cc:225] Handling call
0430 07:59:33.833906 (+ 63541us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b540063cca0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569200900 00007fb569289f22 00007fb569288be0 00007fb569281dda 00007fb56e257b89 00007fb5677c6db4 00007fb5677c6d42 00007fb5677c6c71 00007fb5677c6bec 00007fb5677c5ea2
0430 07:59:34.467975 (+634069us) spinlock_profiling.cc:243] Waited 60.2 ms on lock 0x7b540063cca0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5691fb2a4 00007fb56927ecb7 00007fb56928a41c 00007fb569289a93 00007fb569281dda 00007fb56e257b89 00007fb5677c6db4 00007fb5677c6d42 00007fb5677c6c71 00007fb5677c6bec
0430 07:59:34.471822 (+ 3847us) log.cc:844] Serialized 501 byte log entry
0430 07:59:35.240482 (+768660us) spinlock_profiling.cc:243] Waited 4.97 ms on lock 0x7b180043f600. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661e5b97 00007fb5661e7e01 00007fb5661e7db9 00007fb5661e7d49 00007fb5661e7d11 00007fb5661e700d 0000000000379254 0000000000379099 00007fb5661c8fba
0430 07:59:35.547577 (+307095us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b540063cca0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5691fc522 00007fb5691dd28a 00007fb5691e4133 00007fb5691e40b9 00007fb5691e4049 00007fb5691e4011 00007fb5691e31dd 0000000000379254 0000000000379099 00007fb5661e5ab5
0430 07:59:38.814719 (+3267142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":130724,"raft.queue_time_us":187201,"raft.run_cpu_time_us":67323,"raft.run_wall_time_us":1020523,"spinlock_wait_cycles":239317632,"thread_start_us":38292,"threads_started":1,"wal-append.queue_time_us":331779,"wal-append.run_cpu_time_us":2767,"wal-append.run_wall_time_us":1046232}
I20260430 07:59:39.265637 13015 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:39.319226 9977 leader_election.cc:304] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:39.272348 12866 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.276880 12369 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260430 07:59:39.335521 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:39.336408 9937 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 5.
I20260430 07:59:39.322548 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:39.323487 10008 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 10.
I20260430 07:59:39.301363 12451 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:39.329177 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:39.346405 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:39.330029 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 111181us
I20260430 07:59:39.348286 9903 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:39.338160 10009 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:39.307670 10560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:39.352314 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:39.195120 13006 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.130s
I20260430 07:59:39.352970 9936 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 10.
I20260430 07:59:39.368103 13015 consensus_queue.cc:1048] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260430 07:59:39.323465 12971 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 07:59:39.373747 12799 raft_consensus.cc:2749] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:39.374522 12799 consensus_queue.cc:1048] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260430 07:59:39.380414 9977 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:39.323236 12994 consensus_queue.cc:1048] T cb149eebc54e462c951d45bca72b76e8 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260430 07:59:39.345337 12927 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:39.351414 12992 raft_consensus.cc:2749] T 9031ab0cc4144643b99f584e93b8c75f P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:39.400007 9828 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:39.352777 12451 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:39.346055 12749 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:39.406563 9903 leader_election.cc:304] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:39.411048 9859 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:39.362174 13012 raft_consensus.cc:697] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:39.416316 9931 raft_consensus.cc:1275] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260430 07:59:39.407127 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 87658.8us
I20260430 07:59:39.412499 9903 leader_election.cc:304] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:39.335266 12860 consensus_queue.cc:1059] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260430 07:59:39.414530 9904 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:39.319319 12959 raft_consensus.cc:515] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.335018 12664 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:39.479745 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 101124us
I20260430 07:59:39.406636 12994 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260430 07:59:39.492633 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 64717.4us
I20260430 07:59:39.408965 12619 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.512522 9855 raft_consensus.cc:2393] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:39.513738 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:39.515036 9977 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:59:39.515753 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 111065us
I20260430 07:59:39.522588 10010 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:39.523347 10010 raft_consensus.cc:606] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:39.523751 10010 raft_consensus.cc:614] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:39.399801 13019 consensus_queue.cc:1048] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.096s
I20260430 07:59:39.479632 12369 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:39.528606 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:39.486069 12927 raft_consensus.cc:2749] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:39.426496 12749 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.533311 9903 leader_election.cc:304] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:39.560582 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 63929.5us
W20260430 07:59:39.549063 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:39.560853 9903 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:39.527716 13031 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:39.522680 12812 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:39.530244 12984 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:39.584424 9977 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:59:39.587456 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 66079.1us
I20260430 07:59:39.588232 9828 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:39.588846 9977 leader_election.cc:304] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:39.591790 9977 leader_election.cc:304] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:39.563820 12931 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:39.604797 12931 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:39.496757 9807 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 9fe0ed01a03d4f93a1f8a9644c5497cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 12)
I20260430 07:59:39.605249 12931 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:39.594508 12992 raft_consensus.cc:2764] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
W20260430 07:59:39.605859 9807 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 174765us
I20260430 07:59:39.606376 13034 raft_consensus.cc:2749] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 07:59:39.607342 13034 raft_consensus.cc:2749] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 07:59:39.607363 12959 leader_election.cc:290] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.607898 13036 consensus_queue.cc:1048] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260430 07:59:39.524367 12869 consensus_queue.cc:1059] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.066s
I20260430 07:59:39.596786 13022 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:39.597540 12664 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.558602 13018 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:39.601823 12866 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.572651 12994 raft_consensus.cc:2749] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260430 07:59:39.602875 13009 raft_consensus.cc:2749] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:39.589229 12927 consensus_queue.cc:1048] T 1b60aa0987ed4a9185a3fe724e0acf23 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260430 07:59:39.595582 9977 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
W20260430 07:59:39.609171 12908 consensus_meta.cc:287] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 1.286s user 0.006s sys 0.006s
I20260430 07:59:39.623557 12860 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260430 07:59:39.640277 13028 log.cc:927] Time spent T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260430 07:59:39.639961 12648 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.638338 12784 raft_consensus.cc:2749] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:39.645655 12860 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.647985 12860 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.648826 10003 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:39.649477 10003 raft_consensus.cc:606] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:39.649773 10003 raft_consensus.cc:614] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260430 07:59:39.650032 13032 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260430 07:59:39.651191 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:39.652958 9859 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:39.651962 9930 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:39.651938 12648 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:39.594086 13016 raft_consensus.cc:2749] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260430 07:59:39.610358 12920 raft_consensus.cc:2749] T de66bf2ed62e49aa95b1deff0383d037 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260430 07:59:39.610733 13012 consensus_queue.cc:237] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.673494 9828 leader_election.cc:400] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9.
I20260430 07:59:39.673870 9828 leader_election.cc:403] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260430 07:59:39.596163 12984 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:39.675171 9828 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:39.661518 12369 raft_consensus.cc:697] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [term 8 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:39.681087 13032 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:39.681583 13032 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:39.681926 13032 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:39.683355 12945 raft_consensus.cc:3055] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Stepping down as leader of term 7
W20260430 07:59:39.633769 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 80849.7us
I20260430 07:59:39.638218 12908 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:39.661756 12975 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260430 07:59:39.662886 12955 consensus_meta.cc:287] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.698s user 0.006s sys 0.006s
I20260430 07:59:39.663020 10003 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1"
dest_uuid: "48e81934d52a4e93be5171ceea193032"
mode: GRACEFUL
new_leader_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:55676
I20260430 07:59:39.694994 10003 raft_consensus.cc:606] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Received request to transfer leadership to TS 27f97a330f374849be58a168318c1ec6
I20260430 07:59:39.695501 10003 raft_consensus.cc:614] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260430 07:59:39.664175 12963 consensus_meta.cc:287] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.891s user 0.013s sys 0.000s
I20260430 07:59:39.631671 13018 raft_consensus.cc:515] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.628585 13022 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.677774 12749 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.683954 12945 raft_consensus.cc:740] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:39.660617 12987 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:39.702484 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1900) took 1403 ms. Trace:
I20260430 07:59:39.694491 12975 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:39.693253 10533 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:39.703272 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1894) took 1686 ms. Trace:
I20260430 07:59:39.709043 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:39.709713 9933 raft_consensus.cc:3060] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:39.709259 9861 rpcz_store.cc:276] 0430 07:59:38.016659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:38.016841 (+ 182us) service_pool.cc:225] Handling call
0430 07:59:38.047822 (+ 30981us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:39.145576 (+1097754us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:39.703072 (+557496us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400207954. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:39.703219 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2480980352}
I20260430 07:59:39.708063 9854 rpcz_store.cc:276] 0430 07:59:38.299401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:38.299584 (+ 183us) service_pool.cc:225] Handling call
0430 07:59:38.299943 (+ 359us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:39.207055 (+907112us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:39.702228 (+495173us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b540010cf54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:39.702423 (+ 195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2072622976}
I20260430 07:59:39.697256 10560 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:39.713054 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:39.717765 10003 raft_consensus.cc:1275] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260430 07:59:39.718345 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 157553us
I20260430 07:59:39.719398 9903 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:39.720643 9903 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:39.721905 9903 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:39.722391 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:39.723141 9929 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:39.723258 9903 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:39.724470 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:39.699451 12959 consensus_queue.cc:1048] T 981fd927b4114a73b86c1e232f121c7f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260430 07:59:39.729734 12987 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:39.728096 12860 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:39.727226 10009 raft_consensus.cc:1275] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:39.700268 12963 raft_consensus.cc:515] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.727792 13009 raft_consensus.cc:2804] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:39.727819 13037 consensus_queue.cc:1059] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260430 07:59:39.754012 13037 raft_consensus.cc:3055] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:39.754386 13037 raft_consensus.cc:740] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:39.755211 13037 consensus_queue.cc:260] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.756096 12939 raft_consensus.cc:2749] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:39.756670 13037 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:39.757057 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:39.757557 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:39.757936 9931 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:39.758415 12984 leader_election.cc:290] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:39.759016 13022 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.724750 12927 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:39.729940 8670 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260430 07:59:39.763595 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac5d1b9983db48e198134c6ab5246fbb" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:39.725158 9903 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:39.729507 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:39.733688 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:39.735332 12908 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.731966 13039 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:39.729099 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:39.731263 13025 consensus_queue.cc:1048] T 9031ab0cc4144643b99f584e93b8c75f P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.023s
I20260430 07:59:39.733932 12369 consensus_queue.cc:237] T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.727021 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:39.734788 12945 consensus_queue.cc:260] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.699165 13031 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.016s
I20260430 07:59:39.725346 9936 raft_consensus.cc:3060] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:39.710078 12975 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:39.704566 12619 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260430 07:59:39.748965 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:39.748770 13041 consensus_queue.cc:1059] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260430 07:59:39.751977 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:39.752802 12860 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:39.753170 13009 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:39.699157 12955 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.764645 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:39.938179 10005 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
I20260430 07:59:39.802491 9933 raft_consensus.cc:2468] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 10.
I20260430 07:59:39.830348 9929 raft_consensus.cc:2468] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 11.
W20260430 07:59:39.883744 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 158620us
I20260430 07:59:39.932940 10567 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:48088 (request call id 35) took 2095 ms. Trace:
W20260430 07:59:39.940191 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:39.902644 9863 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37838 (request call id 1944) took 1441 ms. Trace:
I20260430 07:59:39.963378 10008 raft_consensus.cc:3060] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:39.903353 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:39.875459 9931 raft_consensus.cc:2468] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 11.
I20260430 07:59:39.962347 10567 rpcz_store.cc:276] 0430 07:59:37.837939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:37.838092 (+ 153us) service_pool.cc:225] Handling call
0430 07:59:39.932074 (+2093982us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b540063c4d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ed470 00007fb56928c31a 00007fb56e25b11e 00007fb5677d7437 00007fb5677d73c2 00007fb5677d72f1 00007fb5677d726c 00007fb5677d6522 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:39.932765 (+ 691us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3612942592}
I20260430 07:59:39.962289 9903 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:39.749950 9828 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.780s (SENT) [suppressed 24 similar messages]
W20260430 07:59:39.929811 13004 log.cc:927] Time spent T 9fe0ed01a03d4f93a1f8a9644c5497cf P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.980s user 0.006s sys 0.001s
I20260430 07:59:39.933737 12963 leader_election.cc:290] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.968962 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:39.967077 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:39.975541 9854 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 2 FOLLOWER]: Advancing to term 4
I20260430 07:59:39.976080 9933 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 10.
I20260430 07:59:39.963532 9863 rpcz_store.cc:276] 0430 07:59:38.460409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:38.471161 (+ 10752us) service_pool.cc:225] Handling call
0430 07:59:39.901645 (+1430484us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b5400270bd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:39.901805 (+ 160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":378922752}
I20260430 07:59:39.981309 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:39.987404 9858 raft_consensus.cc:2393] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d66db415f0f0402fa431349afbf59888 in current term 6: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:39.990592 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1659) took 1324 ms. Trace:
I20260430 07:59:39.991437 9938 rpcz_store.cc:276] 0430 07:59:38.666107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:39.901763 (+1235656us) service_pool.cc:225] Handling call
0430 07:59:39.990494 (+ 88731us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:39.994279 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 205315us
I20260430 07:59:39.966137 13009 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:39.806996 12799 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:40.032099 13032 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:39.851243 12931 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:39.900104 12451 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.043637 12931 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:40.002399 9901 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:39.994169 12860 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260430 07:59:40.004560 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:40.062086 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 336941us
W20260430 07:59:39.993484 13024 log.cc:927] Time spent T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.601s user 0.000s sys 0.007s
I20260430 07:59:40.032858 12955 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:39.975769 10564 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:40.056663 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 60055.1us
I20260430 07:59:40.082002 9937 raft_consensus.cc:1240] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [term 7 LEADER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20260430 07:59:40.082132 9977 leader_election.cc:336] T c420e10ec8014b4f8df55f542d9693f3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.663s (SENT)
I20260430 07:59:40.044005 12617 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:40.068776 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.046892 9859 raft_consensus.cc:2393] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 5: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:40.020190 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:40.075496 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c10b37fe2e54d7a951545147ac2e381" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:40.047824 12451 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:40.080782 9854 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:40.080859 10012 raft_consensus.cc:1275] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260430 07:59:40.084570 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.095579 9934 raft_consensus.cc:3060] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 3 FOLLOWER]: Advancing to term 4
I20260430 07:59:40.086997 9938 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:40.072571 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.096937 10005 raft_consensus.cc:2393] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 8: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:40.089061 9936 raft_consensus.cc:2468] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:40.079478 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.100500 9929 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:40.102038 10533 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d66db415f0f0402fa431349afbf59888; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:40.103636 13009 raft_consensus.cc:515] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.109704 9932 raft_consensus.cc:1240] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:40.113528 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:40.116763 10567 raft_consensus.cc:2368] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:40.093595 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.119465 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.120226 10006 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 8: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:40.117235 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:40.121312 10007 raft_consensus.cc:2393] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 6: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:40.124902 13032 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:40.125756 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 43654.6us
I20260430 07:59:40.092736 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.127612 10010 raft_consensus.cc:2410] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260430 07:59:40.126801 9977 leader_election.cc:336] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.756s (SENT)
W20260430 07:59:40.130815 9977 leader_election.cc:336] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.866s (SENT)
I20260430 07:59:40.113605 13009 leader_election.cc:290] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:40.131937 9977 leader_election.cc:336] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.880s (SENT)
W20260430 07:59:40.126734 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:40.140544 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.140861 9934 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 4.
I20260430 07:59:40.141361 10007 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 4.
I20260430 07:59:40.143318 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:40.144944 10012 raft_consensus.cc:2393] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 8: Already voted for candidate 48e81934d52a4e93be5171ceea193032 in this term.
I20260430 07:59:40.154861 12930 raft_consensus.cc:993] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: : Instructing follower 27f97a330f374849be58a168318c1ec6 to start an election
I20260430 07:59:40.150995 10533 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:40.155484 13046 raft_consensus.cc:1081] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Signalling peer 27f97a330f374849be58a168318c1ec6 to start an election
W20260430 07:59:40.144959 13043 log.cc:927] Time spent T 2ad545d8bf83435f87a2b1169a24878e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.250s user 0.002s sys 0.001s
I20260430 07:59:40.161616 8670 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260430 07:59:40.162882 8670 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260430 07:59:40.181578 9977 leader_election.cc:336] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.900s (SENT)
I20260430 07:59:40.182168 9977 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:40.184065 9977 leader_election.cc:336] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.956s (SENT)
W20260430 07:59:40.190284 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:40.193173 9938 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 7.
I20260430 07:59:40.193496 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.194577 10004 raft_consensus.cc:2468] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:40.194540 10008 raft_consensus.cc:2468] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:40.196628 10010 raft_consensus.cc:1240] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260430 07:59:40.197594 10533 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:40.155774 13009 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:40.200217 9977 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
I20260430 07:59:40.201937 9935 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:40.205102 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:40.205796 10004 raft_consensus.cc:3060] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:40.209475 9929 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:40.211050 10533 leader_election.cc:304] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:59:40.212630 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 462605us
I20260430 07:59:40.218369 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:40.190099 12931 raft_consensus.cc:2804] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 4 FOLLOWER]: Leader election won for term 4
I20260430 07:59:40.228812 9828 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:40.233971 10009 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260430 07:59:40.155242 12975 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:40.202555 12930 raft_consensus.cc:2804] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:40.219369 12972 raft_consensus.cc:493] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:40.240063 9903 leader_election.cc:304] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:40.220816 12451 consensus_queue.cc:1048] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260430 07:59:40.227856 12942 raft_consensus.cc:2804] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:40.238744 12963 raft_consensus.cc:2804] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election won for term 8
I20260430 07:59:40.238483 12931 raft_consensus.cc:697] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [term 4 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:40.240985 12860 raft_consensus.cc:493] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:40.224012 12920 raft_consensus.cc:2749] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260430 07:59:40.243072 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 59036.8us
I20260430 07:59:40.226401 13032 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.038s
W20260430 07:59:40.279913 9977 leader_election.cc:336] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.965s (SENT)
I20260430 07:59:40.280357 9977 leader_election.cc:304] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:40.224463 13034 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:40.259229 10564 raft_consensus.cc:2468] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:40.260793 12975 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 11 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:40.261348 9828 leader_election.cc:304] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:40.227406 12617 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:40.263360 9903 leader_election.cc:400] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:40.266573 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.295254 9929 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:40.238386 13044 raft_consensus.cc:493] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:40.219638 13009 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.226701 13031 consensus_queue.cc:1048] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260430 07:59:40.257902 13045 raft_consensus.cc:2749] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:40.256810 12930 raft_consensus.cc:697] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 7 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:40.293991 12779 raft_consensus.cc:2749] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260430 07:59:40.324939 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 45049.8us
I20260430 07:59:40.325160 13039 raft_consensus.cc:2749] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205) with higher term. Message: Invalid argument: T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9.
I20260430 07:59:40.299408 10006 raft_consensus.cc:3060] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 6
I20260430 07:59:40.327514 9977 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:40.327827 9856 raft_consensus.cc:1275] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260430 07:59:40.329530 12920 raft_consensus.cc:2804] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:40.329996 12920 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:40.330391 12920 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:40.331849 13039 raft_consensus.cc:493] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:40.332566 13039 raft_consensus.cc:515] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:40.314656 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 53387us
I20260430 07:59:40.334316 9828 leader_election.cc:304] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:40.335273 9828 leader_election.cc:304] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:40.335999 12451 raft_consensus.cc:2804] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 11 FOLLOWER]: Leader election won for term 11
I20260430 07:59:40.336124 9828 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:40.336602 12451 raft_consensus.cc:697] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [term 11 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:40.285996 12984 raft_consensus.cc:2749] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:40.337776 10560 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:40.337657 12451 consensus_queue.cc:237] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.268576 12972 raft_consensus.cc:515] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.339273 10004 raft_consensus.cc:2468] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 7.
I20260430 07:59:40.294606 9903 leader_election.cc:403] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260430 07:59:40.345243 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 86130.5us
I20260430 07:59:40.239259 12955 consensus_queue.cc:1048] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260430 07:59:40.300700 12788 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260430 07:59:40.314173 9929 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 11.
I20260430 07:59:40.317981 13009 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:40.308775 13044 raft_consensus.cc:515] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.323336 12931 consensus_queue.cc:237] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:40.340137 12972 leader_election.cc:290] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:40.282060 12942 raft_consensus.cc:697] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:40.282959 12963 raft_consensus.cc:697] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:40.300014 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:40.297612 13034 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.343161 12930 consensus_queue.cc:237] T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:40.352193 10006 raft_consensus.cc:2468] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:40.355695 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:40.384848 12779 consensus_queue.cc:1048] T a48a2b75c3a34252bf57a4f634457236 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260430 07:59:40.358902 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.303469 12617 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.397926 12617 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:40.344733 9828 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:40.365227 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:40.366237 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.400385 9933 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:40.402096 9933 raft_consensus.cc:1240] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 10. Current term is 11. Ops: [4.3-10.6]
I20260430 07:59:40.361537 13025 raft_consensus.cc:2804] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 11 FOLLOWER]: Leader election won for term 11
I20260430 07:59:40.373993 9930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:40.370970 13006 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260430 07:59:40.376669 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:40.285698 12860 raft_consensus.cc:515] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.379034 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6bdf20bc534fb2a8de1b1c2cbd9af9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:40.410374 9934 raft_consensus.cc:2368] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 9. Current term is 10.
I20260430 07:59:40.380527 9903 leader_election.cc:304] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:40.381444 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:40.412249 9932 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
I20260430 07:59:40.349042 12664 raft_consensus.cc:2804] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Leader election won for term 10
I20260430 07:59:40.385165 10009 raft_consensus.cc:3060] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:40.389984 13034 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:40.371471 10533 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888; no voters:
I20260430 07:59:40.361773 12927 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723) with higher term. Message: Invalid argument: T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for earlier term 6. Current term is 7.
I20260430 07:59:40.372423 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.425520 10007 raft_consensus.cc:2410] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 10 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260430 07:59:40.406106 9930 raft_consensus.cc:2468] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
I20260430 07:59:40.374945 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:40.408267 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.426466 9929 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 9. Current term is 10. Ops: [7.4-9.5]
I20260430 07:59:40.412962 9903 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:40.420346 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 75680.5us
W20260430 07:59:40.423316 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 51967.9us
I20260430 07:59:40.443444 9903 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:40.404513 13009 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:40.409319 13006 raft_consensus.cc:493] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:40.459693 13006 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:40.462110 13009 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.385605 13045 consensus_queue.cc:1048] T 75807667b9164e0ba12c5013baff3b81 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260430 07:59:40.451474 13003 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:40.446168 13034 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:40.454716 12963 consensus_queue.cc:237] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:40.477451 13023 log.cc:927] Time spent T de66bf2ed62e49aa95b1deff0383d037 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.985s user 0.000s sys 0.004s
I20260430 07:59:40.477345 13032 consensus_queue.cc:1059] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260430 07:59:40.479436 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 590) took 1461 ms. Trace:
I20260430 07:59:40.479691 12955 raft_consensus.cc:3055] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Stepping down as leader of term 7
I20260430 07:59:40.480224 12955 raft_consensus.cc:740] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:40.467002 13044 leader_election.cc:290] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:40.421808 12617 raft_consensus.cc:2749] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:40.428956 12664 raft_consensus.cc:697] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 10 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:40.437137 12939 raft_consensus.cc:2804] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 11 FOLLOWER]: Leader election won for term 11
I20260430 07:59:40.425248 12619 raft_consensus.cc:2804] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:40.466392 9932 raft_consensus.cc:1240] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 10. Current term is 11. Ops: [8.5-10.6]
I20260430 07:59:40.448770 9903 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
W20260430 07:59:40.507097 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 58356.3us
W20260430 07:59:40.486931 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:40.510260 9903 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:40.490864 9936 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:40.495982 10009 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 11.
I20260430 07:59:40.479836 9862 rpcz_store.cc:276] 0430 07:59:39.017595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:39.017747 (+ 152us) service_pool.cc:225] Handling call
0430 07:59:39.063100 (+ 45353us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:39.067650 (+ 4550us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:39.067698 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:39.078733 (+ 11035us) log.cc:844] Serialized 26 byte log entry
0430 07:59:39.277798 (+199065us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:39.277850 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:39.277938 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:40.476815 (+1198877us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:40.479103 (+ 2288us) raft_consensus.cc:1685] finished
0430 07:59:40.479126 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:40.479372 (+ 246us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4735,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":17007,"spinlock_wait_cycles":9284992,"thread_start_us":159174,"threads_started":1,"wal-append.queue_time_us":273805}
I20260430 07:59:40.458245 12942 consensus_queue.cc:237] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:40.499099 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 169979us
I20260430 07:59:40.426924 13025 raft_consensus.cc:697] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [term 11 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:40.516698 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.522439 10009 raft_consensus.cc:1275] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:40.531015 10008 raft_consensus.cc:2468] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:40.525226 13006 raft_consensus.cc:515] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.525770 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.532871 10007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.423025 12920 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.542546 10004 raft_consensus.cc:2468] T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:40.543079 9903 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:40.543690 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.466380 13009 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:40.529096 13056 log.cc:927] Time spent T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260430 07:59:40.547238 9859 raft_consensus.cc:1240] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:40.541059 10007 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 7.
I20260430 07:59:40.545007 9903 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:40.551795 9903 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:40.532558 12927 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:40.532831 13045 raft_consensus.cc:2804] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:40.569154 13045 raft_consensus.cc:493] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:40.569866 13045 raft_consensus.cc:3060] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:40.554550 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:40.573063 9861 raft_consensus.cc:2368] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5.
I20260430 07:59:40.554075 12972 raft_consensus.cc:2804] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:40.574864 12972 raft_consensus.cc:493] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:40.575330 12972 raft_consensus.cc:3060] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:40.541496 13034 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:40.540814 12619 raft_consensus.cc:697] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:40.547643 13044 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:40.567265 12920 leader_election.cc:290] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:40.481071 12955 consensus_queue.cc:260] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.532720 12939 raft_consensus.cc:697] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [term 11 LEADER]: Becoming Leader. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:40.561021 12860 leader_election.cc:290] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:40.545790 13041 consensus_queue.cc:1048] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
W20260430 07:59:40.612913 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:40.614151 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 84043.7us
W20260430 07:59:40.628047 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 196062us
I20260430 07:59:40.633080 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.634029 9936 raft_consensus.cc:3060] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:40.635350 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.636132 9931 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:40.601927 13044 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:40.642130 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1238) took 1386 ms. Trace:
I20260430 07:59:40.644565 9937 rpcz_store.cc:276] 0430 07:59:39.242458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:39.402461 (+160003us) service_pool.cc:225] Handling call
0430 07:59:39.427064 (+ 24603us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:40.082436 (+655372us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:40.628655 (+546219us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b54001ee954. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:40.628879 (+ 224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1505344768}
I20260430 07:59:40.606947 13039 leader_election.cc:290] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:40.655977 9828 leader_election.cc:304] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 48e81934d52a4e93be5171ceea193032
W20260430 07:59:40.663214 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 111484us
I20260430 07:59:40.629001 12860 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:40.680652 9828 leader_election.cc:304] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:40.681895 9936 raft_consensus.cc:2468] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 6.
I20260430 07:59:40.683713 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.684365 9937 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:40.582836 13034 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.667054 9929 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.688676 9929 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:40.686098 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1aa1e40099b3420eaada73851145a79e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:40.691293 9931 raft_consensus.cc:2468] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:40.693015 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:40.649725 13044 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
W20260430 07:59:40.687633 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 83745.6us
I20260430 07:59:40.704186 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:40.711681 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 66697.5us
I20260430 07:59:40.712661 9977 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:40.713398 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:40.720045 9931 raft_consensus.cc:2376] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8. Re-sending same reply.
I20260430 07:59:40.684301 12860 raft_consensus.cc:493] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:40.737736 9929 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
I20260430 07:59:40.615216 12664 consensus_queue.cc:237] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:40.745165 9904 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:40.749662 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 69096.5us
W20260430 07:59:40.745033 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:40.712815 12812 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:40.755123 13074 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:40.757822 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:40.764696 9937 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:40.728075 12860 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:40.751513 12959 raft_consensus.cc:2804] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
I20260430 07:59:40.642827 13025 consensus_queue.cc:237] T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.764995 12986 raft_consensus.cc:2804] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:40.766827 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:40.678339 12619 consensus_queue.cc:237] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:40.753613 12939 consensus_queue.cc:237] T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:40.809017 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:40.766739 12812 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:40.846905 9856 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 602) took 1137 ms. Trace:
I20260430 07:59:40.847510 9856 rpcz_store.cc:276] 0430 07:59:39.709026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:39.709361 (+ 335us) service_pool.cc:225] Handling call
0430 07:59:39.711857 (+ 2496us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:40.328529 (+616672us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:40.846010 (+517481us) spinlock_profiling.cc:243] Waited 615 ms on lock 0x7b540010ca54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:40.846800 (+ 790us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1415379200}
I20260430 07:59:40.857623 13062 raft_consensus.cc:2749] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:40.866817 9858 raft_consensus.cc:1240] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:40.876585 9855 raft_consensus.cc:1275] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Refusing update from remote peer 77e19ab653f540339c1c613fad1451b5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260430 07:59:40.877141 9855 pending_rounds.cc:85] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Aborting all ops after (but not including) 4
I20260430 07:59:40.877447 9855 pending_rounds.cc:107] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260430 07:59:40.877741 9855 raft_consensus.cc:2889] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260430 07:59:40.879366 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:40.881865 9858 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:40.883319 9856 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:40.886258 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:40.768573 13006 leader_election.cc:290] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:40.891440 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 174090us
W20260430 07:59:40.901027 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:40.893488 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a48a2b75c3a34252bf57a4f634457236" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:40.902848 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb149eebc54e462c951d45bca72b76e8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:40.946514 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 194838us
I20260430 07:59:40.948544 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:40.949482 10005 raft_consensus.cc:3060] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:40.892127 12858 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:40.962778 10005 raft_consensus.cc:2468] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
I20260430 07:59:40.955394 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:40.968228 9978 leader_election.cc:400] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Vote denied by peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679) with higher term. Message: Invalid argument: T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5.
I20260430 07:59:40.968879 9978 leader_election.cc:403] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260430 07:59:40.973330 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:40.768805 13074 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:40.935339 10012 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.005s sys 0.010s
W20260430 07:59:40.979043 10012 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.006s sys 0.010s
I20260430 07:59:40.977480 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38bcd78c11574f1a9e016f9bb9365ca5" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:40.986905 9855 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:40.995909 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:40.963168 12986 raft_consensus.cc:697] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:40.988569 13088 consensus_queue.cc:1059] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260430 07:59:40.998013 13075 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
I20260430 07:59:40.997356 10005 raft_consensus.cc:2468] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:41.051329 9903 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:41.004258 12945 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:41.063795 10005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:41.008035 12984 consensus_queue.cc:1048] T e18e051a7225424ebb64aedb21bb97df P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260430 07:59:41.012369 13077 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:40.980563 12743 raft_consensus.cc:2749] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:40.989799 13065 log.cc:927] Time spent T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.138s user 0.000s sys 0.001s
I20260430 07:59:40.964485 13034 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:41.045084 9861 raft_consensus.cc:1240] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 7. Current term is 8. Ops: []
I20260430 07:59:41.045614 10012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:40.968274 12858 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.093668 10005 raft_consensus.cc:1240] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9. Ops: []
I20260430 07:59:40.988257 13079 consensus_queue.cc:1048] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.117s
I20260430 07:59:40.989450 12970 consensus_queue.cc:1048] T 9fc86c93a0014c789fda84211b6bd437 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.195s
I20260430 07:59:40.990832 13070 consensus_queue.cc:1059] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.122s
I20260430 07:59:41.082907 13076 raft_consensus.cc:2804] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260430 07:59:41.126708 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 413977us
I20260430 07:59:41.103078 12959 raft_consensus.cc:697] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:41.126248 13080 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:41.095752 12860 raft_consensus.cc:515] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.139729 13074 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:41.136102 13080 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.143167 10011 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:41.120517 13076 raft_consensus.cc:493] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:41.157881 9977 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:41.147054 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 198880us
I20260430 07:59:41.140136 12858 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:41.149713 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 201281us
I20260430 07:59:41.128749 9863 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b"
dest_uuid: "27f97a330f374849be58a168318c1ec6"
from {username='slave'} at 127.0.0.1:41094
I20260430 07:59:41.164816 9863 raft_consensus.cc:493] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260430 07:59:41.165221 9863 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:41.172842 9904 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:41.174925 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:41.178262 13079 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:41.168604 12986 consensus_queue.cc:237] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:41.178964 13079 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.181229 13079 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:41.177445 9828 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:41.185640 13018 leader_election.cc:290] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:41.186936 9977 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:41.187952 13088 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:41.188674 13088 consensus_queue.cc:1048] T 1caba957d05d44788146efd70334d6fe P 48e81934d52a4e93be5171ceea193032 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260430 07:59:41.191545 13081 raft_consensus.cc:2804] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
W20260430 07:59:41.196336 12972 consensus_meta.cc:287] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.613s user 0.010s sys 0.000s
I20260430 07:59:41.196578 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:41.198801 12860 leader_election.cc:290] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:41.188983 9977 leader_election.cc:304] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 77e19ab653f540339c1c613fad1451b5, d66db415f0f0402fa431349afbf59888
W20260430 07:59:41.208047 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:41.165565 13076 raft_consensus.cc:3060] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:41.095500 13077 raft_consensus.cc:515] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.171188 13070 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:41.196846 12972 raft_consensus.cc:515] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.220016 9862 raft_consensus.cc:1240] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [term 10 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 9. Current term is 10. Ops: []
I20260430 07:59:41.222645 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:41.238373 10003 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 9
I20260430 07:59:41.202000 13074 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:41.224162 9863 raft_consensus.cc:515] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:41.241009 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 52109.8us
I20260430 07:59:41.243944 9977 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:41.246043 9977 leader_election.cc:304] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:41.300390 13088 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260430 07:59:41.248426 13045 consensus_meta.cc:287] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.678s user 0.013s sys 0.001s
I20260430 07:59:41.266361 13066 raft_consensus.cc:2749] T 58ea2e276db24da3890f352cef28ff97 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260430 07:59:41.272083 12812 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:41.277899 13081 raft_consensus.cc:697] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:41.276117 12959 consensus_queue.cc:237] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:41.304270 13088 raft_consensus.cc:2749] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260430 07:59:41.331663 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:41.266801 13074 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.315174 13086 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:41.351146 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 165231us
W20260430 07:59:41.364588 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 118878us
I20260430 07:59:41.366526 9828 leader_election.cc:304] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
W20260430 07:59:41.337926 13083 log.cc:927] Time spent T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.161s user 0.001s sys 0.001s
I20260430 07:59:41.376243 9977 leader_election.cc:400] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147) with higher term. Message: Invalid argument: T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 9. Current term is 10.
W20260430 07:59:41.377390 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:41.381978 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:41.382863 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:41.395009 9857 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 6. Current term is 7. Ops: []
W20260430 07:59:41.398480 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 170322us
I20260430 07:59:41.401443 9828 leader_election.cc:304] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:41.402935 9857 raft_consensus.cc:1240] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:41.409967 9828 leader_election.cc:304] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: 48e81934d52a4e93be5171ceea193032
I20260430 07:59:41.411727 9828 leader_election.cc:304] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:41.396152 12812 raft_consensus.cc:2804] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election won for term 9
I20260430 07:59:41.413179 9828 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:41.412657 9977 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:41.415019 9977 leader_election.cc:304] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:41.355337 13045 raft_consensus.cc:515] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.397143 13088 consensus_queue.cc:1059] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260430 07:59:41.400774 13025 consensus_queue.cc:1059] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260430 07:59:41.413147 12664 raft_consensus.cc:2804] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Leader election won for term 6
I20260430 07:59:41.450366 9929 raft_consensus.cc:1240] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260430 07:59:41.369922 13086 raft_consensus.cc:515] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:41.415933 12970 raft_consensus.cc:2804] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader election won for term 7
W20260430 07:59:41.460361 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:41.464226 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 49266.3us
I20260430 07:59:41.454679 12972 leader_election.cc:290] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:41.418466 12451 raft_consensus.cc:2804] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election won for term 5
I20260430 07:59:41.466123 9977 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:59:41.529719 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 63662.3us
I20260430 07:59:41.447227 13099 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:41.562819 9977 leader_election.cc:304] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:41.565698 10004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
W20260430 07:59:41.478796 13102 log.cc:927] Time spent T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
W20260430 07:59:41.480792 13084 log.cc:927] Time spent T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.339s user 0.001s sys 0.000s
W20260430 07:59:41.486665 13101 log.cc:927] Time spent T 3028e67ff7d948ee8accee91fbcb70c3 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
W20260430 07:59:41.474078 13100 log.cc:927] Time spent T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
W20260430 07:59:41.486238 13044 consensus_meta.cc:287] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.766s user 0.014s sys 0.000s
I20260430 07:59:41.453183 13080 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:41.570750 9855 raft_consensus.cc:1240] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Rejecting Update request from peer 77e19ab653f540339c1c613fad1451b5 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:41.564183 12664 raft_consensus.cc:697] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
W20260430 07:59:41.586437 13107 log.cc:927] Time spent T ac5d1b9983db48e198134c6ab5246fbb P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.154s user 0.000s sys 0.000s
W20260430 07:59:41.569792 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 156349us
I20260430 07:59:41.464990 13074 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:41.527861 12779 raft_consensus.cc:2749] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:41.537710 13066 raft_consensus.cc:3055] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 10 LEADER]: Stepping down as leader of term 10
I20260430 07:59:41.571237 12451 raft_consensus.cc:697] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:41.483287 13113 raft_consensus.cc:2804] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:41.476459 13092 raft_consensus.cc:3055] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:41.570636 13086 leader_election.cc:290] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:41.589377 13094 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:41.601441 13094 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:41.604532 13094 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:41.422129 13081 consensus_queue.cc:237] T eaaca7c934d147a396bbaee7e10d6496 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.587514 12664 consensus_queue.cc:237] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:41.591064 12945 consensus_meta.cc:287] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.509s user 0.012s sys 0.000s
I20260430 07:59:41.485369 13098 consensus_queue.cc:1059] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.105s
I20260430 07:59:41.483780 13104 consensus_queue.cc:1059] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.109s
I20260430 07:59:41.602141 13109 raft_consensus.cc:2804] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 07:59:41.536617 12812 raft_consensus.cc:697] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:41.616683 13109 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:41.617493 13109 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:41.617360 12812 consensus_queue.cc:237] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:41.616272 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 53520.9us
I20260430 07:59:41.628325 13109 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.593629 13066 raft_consensus.cc:740] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:41.596462 13113 raft_consensus.cc:493] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:41.597605 13092 raft_consensus.cc:740] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:41.593011 13088 consensus_queue.cc:1059] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260430 07:59:41.653370 13117 consensus_queue.cc:1059] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:41.675674 10003 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 9.
I20260430 07:59:41.581369 13044 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.645043 13113 raft_consensus.cc:3060] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:41.655258 12970 raft_consensus.cc:697] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:41.688395 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:41.698689 10003 raft_consensus.cc:2468] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:41.702105 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:41.702616 13086 raft_consensus.cc:3055] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 10 LEADER]: Stepping down as leader of term 10
I20260430 07:59:41.703521 13086 raft_consensus.cc:740] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:41.703681 12970 consensus_queue.cc:237] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.702481 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:41.709019 9857 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260430 07:59:41.707482 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 55031.3us
I20260430 07:59:41.743664 9828 leader_election.cc:304] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:41.709542 10009 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1926) took 1168 ms. Trace:
I20260430 07:59:41.744601 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:41.745462 10009 rpcz_store.cc:276] 0430 07:59:40.541631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.541856 (+ 225us) service_pool.cc:225] Handling call
0430 07:59:40.555260 (+ 13404us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:40.578164 (+ 22904us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:40.578215 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:40.578845 (+ 630us) log.cc:844] Serialized 26 byte log entry
0430 07:59:40.898619 (+319774us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:40.899542 (+ 923us) log.cc:844] Serialized 14 byte log entry
0430 07:59:40.900548 (+ 1006us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:40.900628 (+ 80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:41.479820 (+579192us) spinlock_profiling.cc:243] Waited 7.89 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:41.699967 (+220147us) raft_consensus.cc:1685] finished
0430 07:59:41.709261 (+ 9294us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:41.709469 (+ 208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9943,"prepare.queue_time_us":149,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":20729,"raft.run_cpu_time_us":462,"raft.run_wall_time_us":42908,"spinlock_wait_cycles":18141568,"thread_start_us":254170,"threads_started":1,"wal-append.queue_time_us":529018}
I20260430 07:59:41.762241 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1964) took 1188 ms. Trace:
I20260430 07:59:41.745806 13031 raft_consensus.cc:493] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:41.761541 10003 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:41.762682 9854 rpcz_store.cc:276] 0430 07:59:40.573621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.575874 (+ 2253us) service_pool.cc:225] Handling call
0430 07:59:40.576193 (+ 319us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:40.576413 (+ 220us) raft_consensus.cc:1534] Early marking committed up to index 6
0430 07:59:40.583149 (+ 6736us) log.cc:844] Serialized 14 byte log entry
0430 07:59:41.591612 (+1008463us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b180026f2c0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb5661c8587 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e5242 00007fb568e1c5ab 00007fb568e220c3 00007fb568e225da 00007fb56928de0a
0430 07:59:41.745586 (+153974us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:41.745638 (+ 52us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:41.745693 (+ 55us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:41.762048 (+ 16355us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:41.762180 (+ 132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6235,"prepare.queue_time_us":5792,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"spinlock_wait_cycles":42696960,"thread_start_us":147424,"threads_started":1}
I20260430 07:59:41.710776 9937 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260430 07:59:41.766022 10567 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:41.686390 13045 leader_election.cc:290] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:41.754504 13070 consensus_meta.cc:287] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.505s user 0.006s sys 0.005s
I20260430 07:59:41.706410 13086 consensus_queue.cc:260] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.778214 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:41.772347 13031 raft_consensus.cc:515] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.793828 10009 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:41.794742 10009 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 8.
I20260430 07:59:41.795542 13086 raft_consensus.cc:3060] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:41.797474 13070 raft_consensus.cc:2749] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 07:59:41.798244 10004 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:41.803135 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:41.803700 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 60128us
I20260430 07:59:41.804520 9828 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:41.808161 10004 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 48e81934d52a4e93be5171ceea193032 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 11.
I20260430 07:59:41.839254 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:41.774075 13111 raft_consensus.cc:2749] T 19c3ca044c3d4377b31c4d07c513415c P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:41.853483 9863 leader_election.cc:290] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:41.857535 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 53084.9us
I20260430 07:59:41.767414 13077 leader_election.cc:290] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:41.858357 9828 leader_election.cc:304] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:41.689663 12451 consensus_queue.cc:237] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.824839 13080 raft_consensus.cc:2804] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:41.708144 13076 raft_consensus.cc:515] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.875173 9828 leader_election.cc:304] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, d66db415f0f0402fa431349afbf59888
I20260430 07:59:41.893234 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41094 (request call id 611) took 1816 ms. Trace:
I20260430 07:59:41.904321 9828 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:41.905138 9860 rpcz_store.cc:276] 0430 07:59:40.076474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.874141 (+797667us) service_pool.cc:225] Handling call
0430 07:59:40.874537 (+ 396us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:40.889122 (+ 14585us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:40.889153 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:40.889199 (+ 46us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:40.889230 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:41.892473 (+1003243us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b540029ca54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:41.892525 (+ 52us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:41.892753 (+ 228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33073024}
I20260430 07:59:41.709440 13092 consensus_queue.cc:260] T 1aa1e40099b3420eaada73851145a79e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.923094 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d66db415f0f0402fa431349afbf59888"
I20260430 07:59:41.875138 13080 raft_consensus.cc:493] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:41.728467 13066 consensus_queue.cc:260] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.885764 13132 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:41.892452 13099 raft_consensus.cc:2804] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260430 07:59:41.909535 13075 consensus_meta.cc:287] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.827s user 0.011s sys 0.004s
I20260430 07:59:41.934041 10564 raft_consensus.cc:3055] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Stepping down as leader of term 6
W20260430 07:59:41.959203 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:41.940373 13105 raft_consensus.cc:3055] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Stepping down as leader of term 8
I20260430 07:59:41.917128 13025 raft_consensus.cc:2749] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:41.914667 13125 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:41.914525 13135 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:41.908453 13109 leader_election.cc:290] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:41.946405 10009 raft_consensus.cc:1275] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260430 07:59:41.968999 10003 raft_consensus.cc:1240] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260430 07:59:41.933203 13080 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:41.944222 13044 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:41.949338 13074 raft_consensus.cc:2804] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260430 07:59:41.984714 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:41.967546 13132 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:41.966568 13075 raft_consensus.cc:2749] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679) with higher term. Message: Invalid argument: T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for earlier term 4. Current term is 5.
I20260430 07:59:41.996884 13132 leader_election.cc:290] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:41.967572 13105 raft_consensus.cc:740] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:41.998664 13105 consensus_queue.cc:260] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:41.999718 13105 raft_consensus.cc:3060] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:42.002925 13138 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:42.003598 13138 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:42.008155 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 835103us
I20260430 07:59:42.013338 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:41.954432 10564 raft_consensus.cc:740] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d66db415f0f0402fa431349afbf59888, State: Running, Role: LEADER
I20260430 07:59:42.013876 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7e202af141f43f7b88e4231eed0ba94" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:42.014994 9857 raft_consensus.cc:3060] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:42.016127 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1261) took 1306 ms. Trace:
I20260430 07:59:42.016775 9934 rpcz_store.cc:276] 0430 07:59:40.709406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.851516 (+142110us) service_pool.cc:225] Handling call
0430 07:59:40.851917 (+ 401us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:40.852155 (+ 238us) raft_consensus.cc:1534] Early marking committed up to index 4
0430 07:59:40.858558 (+ 6403us) log.cc:844] Serialized 14 byte log entry
0430 07:59:41.167268 (+308710us) log.cc:844] Serialized 14 byte log entry
0430 07:59:42.004839 (+837571us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b18004e31c0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb566159e1a 00007fb56f7f25c9 00007fb5660181f0 00007fb5661e2b24 00007fb5661e2a95 00007fb56927b440 00007fb56928daf6 00007fb56929e667 00007fb56929e5e8
0430 07:59:42.005102 (+ 263us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:42.005181 (+ 79us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:42.005251 (+ 70us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.015885 (+ 10634us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:42.016051 (+ 166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4526,"prepare.queue_time_us":5129,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":99,"raft.queue_time_us":844888,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":30508,"spinlock_wait_cycles":30607104,"thread_start_us":307962,"threads_started":1,"wal-append.queue_time_us":446861}
I20260430 07:59:42.022342 10012 raft_consensus.cc:2468] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:42.022483 10567 raft_consensus.cc:1275] T 7c10b37fe2e54d7a951545147ac2e381 P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260430 07:59:41.970692 13099 raft_consensus.cc:493] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
W20260430 07:59:42.033882 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 829330us
I20260430 07:59:42.039371 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:42.045655 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:42.039067 12963 raft_consensus.cc:493] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:42.054306 12963 raft_consensus.cc:515] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.038676 12860 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:42.060935 13113 raft_consensus.cc:515] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.063580 10009 raft_consensus.cc:1240] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260430 07:59:42.064733 13113 leader_election.cc:290] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:42.065560 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddad6955cd64efd850fddc61341be7e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:42.075523 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1256) took 1397 ms. Trace:
I20260430 07:59:42.076236 9936 rpcz_store.cc:276] 0430 07:59:40.678436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.682862 (+ 4426us) service_pool.cc:225] Handling call
0430 07:59:40.686287 (+ 3425us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:40.686578 (+ 291us) raft_consensus.cc:1534] Early marking committed up to index 6
0430 07:59:41.068094 (+381516us) log.cc:844] Serialized 14 byte log entry
0430 07:59:42.075001 (+1006907us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:42.075086 (+ 85us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:42.075154 (+ 68us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.075259 (+ 105us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:42.075443 (+ 184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":780760,"prepare.queue_time_us":20868,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"thread_start_us":547292,"threads_started":2,"wal-append.queue_time_us":326976}
I20260430 07:59:42.103015 9934 raft_consensus.cc:1240] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260430 07:59:42.103350 9936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 458) took 1302 ms. Trace:
I20260430 07:59:42.103636 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1263) took 1378 ms. Trace:
I20260430 07:59:42.104113 9934 rpcz_store.cc:276] 0430 07:59:40.725571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:42.020436 (+1294865us) service_pool.cc:225] Handling call
0430 07:59:42.102841 (+ 82405us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:42.103445 (+ 604us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.103592 (+ 147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:41.991698 13074 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:42.105929 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:42.103924 9936 rpcz_store.cc:276] 0430 07:59:40.801181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:42.078561 (+1277380us) service_pool.cc:225] Handling call
0430 07:59:42.102670 (+ 24109us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:42.102904 (+ 234us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:42.102943 (+ 39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:42.103007 (+ 64us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:42.103054 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.103122 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:42.103272 (+ 150us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260430 07:59:42.111601 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 68532.1us
I20260430 07:59:42.114457 9859 raft_consensus.cc:1240] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [term 6 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:42.115623 9932 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1252) took 1481 ms. Trace:
I20260430 07:59:42.125082 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:42.128463 9930 raft_consensus.cc:1240] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260430 07:59:42.129314 9862 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1960) took 1599 ms. Trace:
I20260430 07:59:42.130043 9862 rpcz_store.cc:276] 0430 07:59:40.529545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.541565 (+ 12020us) service_pool.cc:225] Handling call
0430 07:59:40.557311 (+ 15746us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:41.241541 (+684230us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.128970 (+887429us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x7b54002710d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:42.129212 (+ 242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1516030720}
I20260430 07:59:42.133986 9857 raft_consensus.cc:2468] T a7e202af141f43f7b88e4231eed0ba94 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 9.
I20260430 07:59:42.138484 9936 raft_consensus.cc:1275] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260430 07:59:42.140311 9936 raft_consensus.cc:1275] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260430 07:59:42.033447 13099 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
W20260430 07:59:42.147406 9931 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35268 (request call id 1258) took 1441 ms (client timeout 1780 ms). Trace:
W20260430 07:59:42.147918 9931 rpcz_store.cc:269] 0430 07:59:40.687367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.692684 (+ 5317us) service_pool.cc:225] Handling call
0430 07:59:42.128056 (+1435372us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b54003241d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:42.128231 (+ 175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59445632}
W20260430 07:59:42.147635 10006 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:53754 (request call id 492) took 1906 ms (client timeout 2246 ms). Trace:
I20260430 07:59:42.149567 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9001555606564c6081e62a65ffa52dc8" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:42.149847 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58726 (request call id 459) took 1261 ms. Trace:
W20260430 07:59:42.150840 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:42.150767 9934 rpcz_store.cc:276] 0430 07:59:40.888579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:42.104983 (+1216404us) service_pool.cc:225] Handling call
0430 07:59:42.149122 (+ 44139us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:42.149452 (+ 330us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:42.149482 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:42.149533 (+ 51us) raft_consensus.cc:1645] Marking committed up to 6
0430 07:59:42.149565 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.149622 (+ 57us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:42.149796 (+ 174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:42.101419 12963 leader_election.cc:290] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:41.986279 13135 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.062251 12860 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.062018 10564 consensus_queue.cc:260] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } attrs { replace: true } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:42.189513 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 263150us
I20260430 07:59:42.150568 9931 raft_consensus.cc:3060] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:42.203748 10566 raft_consensus.cc:1275] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260430 07:59:42.154080 9936 raft_consensus.cc:1275] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:42.333410 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.116194 9932 rpcz_store.cc:276] 0430 07:59:40.633949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.634221 (+ 272us) service_pool.cc:225] Handling call
0430 07:59:40.661572 (+ 27351us) raft_consensus.cc:1471] Updating replica for 2 ops
0430 07:59:40.684291 (+ 22719us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:40.684328 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0430 07:59:40.705939 (+ 21611us) log.cc:844] Serialized 52 byte log entry
0430 07:59:41.339091 (+633152us) spinlock_profiling.cc:243] Waited 6.93 ms on lock 0x7b180045cc40. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56f04c251 00007fb56f058ab1 00007fb568e2a85c 00007fb5661c8c0b 00007fb56efc1bc9 00007fb5661e4cdb 00007fb5661e2f7f 00007fb5661e5242 00007fb568e1c5ab 00007fb568e220c3 00007fb568e22471
0430 07:59:41.339308 (+ 217us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:41.373197 (+ 33889us) log.cc:844] Serialized 14 byte log entry
0430 07:59:42.101936 (+728739us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.102021 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:42.102806 (+ 785us) raft_consensus.cc:1685] finished
0430 07:59:42.115176 (+ 12370us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:42.115563 (+ 387us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":728413,"prepare.queue_time_us":33767,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":3822,"raft.queue_time_us":798550,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":4352,"spinlock_wait_cycles":15945984,"thread_start_us":32242,"threads_started":1,"wal-append.queue_time_us":181253}
I20260430 07:59:42.151479 9854 raft_consensus.cc:1240] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [term 7 LEADER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:42.335575 9903 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:42.153473 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:42.305706 9977 leader_election.cc:336] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 2.005s (SENT)
W20260430 07:59:42.149461 10006 rpcz_store.cc:269] 0430 07:59:40.217832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.218014 (+ 182us) service_pool.cc:225] Handling call
0430 07:59:42.123797 (+1905783us) spinlock_profiling.cc:243] Waited 78.1 ms on lock 0x7b540004a1d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:42.123977 (+ 180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":179703040}
I20260430 07:59:42.337672 9977 leader_election.cc:304] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:42.105228 13018 raft_consensus.cc:2749] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260430 07:59:42.348232 10534 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.005s sys 0.013s
W20260430 07:59:42.348826 10534 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s user 0.005s sys 0.013s
I20260430 07:59:42.115645 12945 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:42.350826 9936 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:42.352665 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:42.212270 13138 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:42.339745 9904 leader_election.cc:304] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:42.116086 13074 raft_consensus.cc:3060] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:42.142674 13037 raft_consensus.cc:3055] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Stepping down as leader of term 5
W20260430 07:59:42.361887 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:42.362583 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.155385 13044 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:42.179716 13111 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:42.367705 13149 consensus_queue.cc:1048] T cd7ed510277642ca8409dce1775947c1 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260430 07:59:42.201761 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:42.372507 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 66850.3us
I20260430 07:59:42.339460 13039 raft_consensus.cc:493] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:42.201911 13045 raft_consensus.cc:493] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:42.188731 13113 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:42.181818 12661 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:42.202875 13041 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:42.366034 9828 leader_election.cc:304] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:42.388813 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
W20260430 07:59:42.388885 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 49236.1us
I20260430 07:59:42.198683 12858 raft_consensus.cc:493] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:42.391443 9932 raft_consensus.cc:2393] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:42.338974 13076 leader_election.cc:290] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:42.392100 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:42.367234 9931 raft_consensus.cc:2468] T 9001555606564c6081e62a65ffa52dc8 P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 11.
W20260430 07:59:42.397405 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:42.403851 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76d5ae1122e343f28f49a7fdbf3db843" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:42.142067 13125 consensus_queue.cc:1048] T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.267s
I20260430 07:59:42.441045 13142 raft_consensus.cc:2749] T a7e202af141f43f7b88e4231eed0ba94 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:42.405558 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fc86c93a0014c789fda84211b6bd437" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:42.365016 10564 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:42.528301 12858 raft_consensus.cc:515] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.483467 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1260) took 1779 ms. Trace:
W20260430 07:59:42.443318 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 55397.6us
I20260430 07:59:42.496426 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:42.529206 10006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:42.539512 10006 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:42.527259 13041 raft_consensus.cc:515] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:42.532845 9935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1662) took 3853 ms (client timeout 2005 ms). Trace:
I20260430 07:59:42.531666 9937 rpcz_store.cc:276] 0430 07:59:40.703970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.820104 (+116134us) service_pool.cc:225] Handling call
0430 07:59:40.868126 (+ 48022us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:41.765990 (+897864us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.483099 (+717109us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b54001b26d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:42.483385 (+ 286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1936651648}
W20260430 07:59:42.539472 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 173486us
W20260430 07:59:42.549801 9935 rpcz_store.cc:269] 0430 07:59:38.679410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:38.679546 (+ 136us) service_pool.cc:225] Handling call
0430 07:59:42.532651 (+3853105us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540003c354. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:42.532776 (+ 125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3347424128}
W20260430 07:59:42.540437 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:42.530040 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2df038061ff495dbdb2eaa4ad73c76e" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.541646 9904 leader_election.cc:304] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:42.536190 10003 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 5.
I20260430 07:59:42.534144 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd7ed510277642ca8409dce1775947c1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.552750 9932 raft_consensus.cc:2468] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
W20260430 07:59:42.559159 9828 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:42.556322 9904 leader_election.cc:304] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:42.562179 10003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:42.564291 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.567695 9932 raft_consensus.cc:2393] T 18a52542f8aa4aeca8e319d4f60e9699 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 8: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
W20260430 07:59:42.483668 13145 log.cc:927] Time spent T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260430 07:59:42.578372 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.390960 13037 raft_consensus.cc:740] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:42.575994 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db24b74e2fa44ae2beba675582424a3b" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
W20260430 07:59:42.460080 13090 log.cc:927] Time spent T 5cfb45b2ec3c45c38f2aba832ed29742 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 1.321s user 0.008s sys 0.015s
I20260430 07:59:42.460685 13009 raft_consensus.cc:2804] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election won for term 9
I20260430 07:59:42.486320 13143 raft_consensus.cc:493] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:42.595594 9937 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b978c5a9ab41402c806352f5ace14295" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.556309 12986 raft_consensus.cc:2804] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 11 FOLLOWER]: Leader election won for term 11
I20260430 07:59:42.590101 9931 raft_consensus.cc:2468] T db24b74e2fa44ae2beba675582424a3b P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:42.610456 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.611217 9931 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 5.
I20260430 07:59:42.612260 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.612426 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:42.516297 12927 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260430 07:59:42.555979 13142 raft_consensus.cc:493] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:42.487145 13135 leader_election.cc:290] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:42.391888 13081 consensus_queue.cc:1048] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.201s
I20260430 07:59:42.593500 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8dad8acea649479d1e927cd32599fe" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:42.461141 12860 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:42.596745 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.597296 9932 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.539211 13120 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:42.633508 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1307) took 1928 ms. Trace:
I20260430 07:59:42.553591 13045 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.547808 13062 raft_consensus.cc:3055] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Stepping down as leader of term 6
I20260430 07:59:42.547852 13113 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.588920 9862 raft_consensus.cc:1240] T 7303f2c98e1b411f83314a2c2f146d4b P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Rejecting Update request from peer d66db415f0f0402fa431349afbf59888 for earlier term 6. Current term is 7. Ops: []
W20260430 07:59:42.570580 12930 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.173s user 0.009s sys 0.013s
I20260430 07:59:42.604734 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:42.365376 12945 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.614163 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.538831 13099 raft_consensus.cc:515] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.403153 12970 consensus_queue.cc:1059] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.198s
I20260430 07:59:42.590147 13149 raft_consensus.cc:2749] T cb149eebc54e462c951d45bca72b76e8 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260430 07:59:42.532303 13080 raft_consensus.cc:515] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.505380 13134 raft_consensus.cc:493] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:42.427001 13044 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:42.603238 13105 consensus_meta.cc:287] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.603s user 0.012s sys 0.000s
I20260430 07:59:42.604380 12798 raft_consensus.cc:493] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:42.549042 12661 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.613648 13009 raft_consensus.cc:697] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [term 9 LEADER]: Becoming Leader. State: Replica: 77e19ab653f540339c1c613fad1451b5, State: Running, Role: LEADER
I20260430 07:59:42.631767 12858 leader_election.cc:290] T 725a25b9eaa049c3ba8535dbe1100561 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:42.615639 12927 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:42.632114 9977 leader_election.cc:304] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:42.574311 13039 raft_consensus.cc:515] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.427570 13111 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.633927 10008 rpcz_store.cc:276] 0430 07:59:40.705344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.705486 (+ 142us) service_pool.cc:225] Handling call
0430 07:59:40.705897 (+ 411us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:40.716611 (+ 10714us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:40.716638 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:40.729637 (+ 12999us) log.cc:844] Serialized 26 byte log entry
0430 07:59:40.980828 (+251191us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:40.980878 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:40.980945 (+ 67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:40.999612 (+ 18667us) spinlock_profiling.cc:243] Waited 7.13 ms on lock 0x7b4800250158. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb56e1bb1a1 00007fb56e1b2237 00007fb56e1b20ca 00007fb56e2a992f 00007fb56e2a95f7 00007fb56e2bdc1c 00007fb56e2bdbc8 00007fb56e2bdb48 00007fb56e2bdb03 00007fb56e2bcdac 00007fb56d1e5b36
0430 07:59:42.359307 (+1359695us) spinlock_profiling.cc:243] Waited 8.32 ms on lock 0x7b440002534c. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5660f2375 00007fb5660f2d92 00007fb568e2e57b 00007fb568e29e4f 00007fb568e1dc0e 00007fb568e1cf56 00007fb568e1cb78 00007fb568e265c1 00007fb568e26579 00007fb568e26509
0430 07:59:42.633266 (+273959us) raft_consensus.cc:1685] finished
0430 07:59:42.633308 (+ 42us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:42.633450 (+ 142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10321,"raft.queue_time_us":49999,"raft.run_cpu_time_us":2457,"raft.run_wall_time_us":241263,"spinlock_wait_cycles":35526272,"thread_start_us":209711,"threads_started":1,"wal-append.queue_time_us":355657}
I20260430 07:59:42.600273 9935 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "452aa377ba5f47c3bdd4bf5d4e48059d" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.601302 9937 raft_consensus.cc:2468] T b978c5a9ab41402c806352f5ace14295 P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 4.
W20260430 07:59:42.636160 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 300614us
I20260430 07:59:42.632725 13160 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:42.570870 12920 raft_consensus.cc:493] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:42.645062 9934 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7303f2c98e1b411f83314a2c2f146d4b" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:42.671093 10006 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
W20260430 07:59:42.641062 12930 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.244s user 0.009s sys 0.014s
I20260430 07:59:42.632010 13041 leader_election.cc:290] T 981fd927b4114a73b86c1e232f121c7f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:42.713631 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44010 (request call id 1914) took 2571 ms. Trace:
I20260430 07:59:42.641048 13062 raft_consensus.cc:740] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 48e81934d52a4e93be5171ceea193032, State: Running, Role: LEADER
I20260430 07:59:42.630014 13142 raft_consensus.cc:515] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.614595 13143 raft_consensus.cc:515] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.713984 13134 raft_consensus.cc:515] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.848488 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5cfb45b2ec3c45c38f2aba832ed29742" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:42.848608 10003 raft_consensus.cc:2468] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 6.
I20260430 07:59:42.849644 9934 raft_consensus.cc:3060] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:42.854554 9935 raft_consensus.cc:2468] T 452aa377ba5f47c3bdd4bf5d4e48059d P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 7.
I20260430 07:59:42.855312 9931 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:42.847839 13062 consensus_queue.cc:260] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.728552 9936 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
W20260430 07:59:42.854765 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 298421us
W20260430 07:59:42.859773 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 227664us
I20260430 07:59:42.637493 13120 raft_consensus.cc:515] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.863449 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 508) took 1292 ms. Trace:
I20260430 07:59:42.854487 10010 rpcz_store.cc:276] 0430 07:59:40.135025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.135170 (+ 145us) service_pool.cc:225] Handling call
0430 07:59:40.196404 (+ 61234us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:40.198210 (+ 1806us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.705699 (+2507489us) spinlock_profiling.cc:243] Waited 36.6 ms on lock 0x7b54002c4550. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:42.705978 (+ 279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84210048}
I20260430 07:59:42.899871 9904 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:42.899605 10008 rpcz_store.cc:276] 0430 07:59:41.571291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:42.862528 (+1291237us) service_pool.cc:225] Handling call
0430 07:59:42.862787 (+ 259us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:42.863002 (+ 215us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:42.863040 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:42.863115 (+ 75us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:42.863162 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.863236 (+ 74us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:42.863375 (+ 139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:42.900067 9977 leader_election.cc:304] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888
I20260430 07:59:42.692931 13076 raft_consensus.cc:2749] T 38bcd78c11574f1a9e016f9bb9365ca5 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:42.904750 9904 leader_election.cc:304] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
W20260430 07:59:42.778285 13115 log.cc:927] Time spent T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260430 07:59:42.905660 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:42.902967 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 45645.1us
I20260430 07:59:42.669679 13037 consensus_queue.cc:260] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.915474 9903 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:42.765031 13149 raft_consensus.cc:493] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:42.713984 12369 raft_consensus.cc:493] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:42.761320 13105 raft_consensus.cc:3055] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Stepping down as leader of term 9
I20260430 07:59:42.778023 13136 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:42.777782 13151 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:42.800550 12986 raft_consensus.cc:697] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [term 11 LEADER]: Becoming Leader. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:42.936638 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 510) took 1341 ms. Trace:
I20260430 07:59:42.857182 13081 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:42.948556 10010 rpcz_store.cc:276] 0430 07:59:41.595142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:42.899871 (+1304729us) service_pool.cc:225] Handling call
0430 07:59:42.936111 (+ 36240us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:42.936305 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:42.936333 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:42.936380 (+ 47us) raft_consensus.cc:1645] Marking committed up to 7
0430 07:59:42.936411 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:42.936465 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:42.936579 (+ 114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:42.905117 13041 raft_consensus.cc:493] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
W20260430 07:59:42.956147 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.132:40723, user_credentials={real_user=slave}} blocked reactor thread for 56063.8us
I20260430 07:59:42.908456 12970 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:42.973654 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:42.965437 13135 raft_consensus.cc:493] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:42.937649 13149 raft_consensus.cc:515] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:42.968096 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:42.972347 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
W20260430 07:59:42.964840 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 281109us
I20260430 07:59:42.740957 12798 raft_consensus.cc:515] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.921205 13161 raft_consensus.cc:493] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:42.938591 13105 raft_consensus.cc:740] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:42.897934 12920 raft_consensus.cc:515] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.943009 13062 raft_consensus.cc:3060] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Advancing to term 7
I20260430 07:59:42.956305 13037 raft_consensus.cc:3060] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:42.952948 13076 raft_consensus.cc:2749] T bddad6955cd64efd850fddc61341be7e P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260430 07:59:42.922216 13157 raft_consensus.cc:2749] T 829ed212483c445ba0dd4ae328bde3c2 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260430 07:59:42.991071 9978 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:42.993167 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "725a25b9eaa049c3ba8535dbe1100561" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.006819 9860 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.007721 9860 raft_consensus.cc:2468] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 9.
W20260430 07:59:42.979564 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 64218.9us
I20260430 07:59:43.009766 9903 leader_election.cc:304] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:42.985692 9934 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:43.012651 9934 raft_consensus.cc:1240] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 6. Current term is 7. Ops: []
I20260430 07:59:43.016453 10008 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [term 11 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260430 07:59:42.944784 12369 raft_consensus.cc:515] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:43.015367 9903 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:43.019980 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "981fd927b4114a73b86c1e232f121c7f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:42.995945 12955 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:42.995395 13150 raft_consensus.cc:2749] T a48a2b75c3a34252bf57a4f634457236 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260430 07:59:43.026813 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:43.026422 9903 leader_election.cc:304] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:42.901247 13160 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:42.977829 13041 raft_consensus.cc:515] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.040597 13080 leader_election.cc:290] T 15f2cd3a47984929840836a88ecfd8b0 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:43.081257 13158 log.cc:927] Time spent T 15f2cd3a47984929840836a88ecfd8b0 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.172s user 0.001s sys 0.000s
I20260430 07:59:43.081718 13134 leader_election.cc:290] T a66ec69e8f2c493fa90c40e25719d298 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.082353 13134 raft_consensus.cc:2749] T 2e8dad8acea649479d1e927cd32599fe P 48e81934d52a4e93be5171ceea193032 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260430 07:59:42.977126 13151 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:43.028963 13174 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260430 07:59:43.085467 9932 raft_consensus.cc:2468] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 4.
I20260430 07:59:43.086112 9934 raft_consensus.cc:1275] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260430 07:59:43.023244 13076 raft_consensus.cc:2749] T 9fc86c93a0014c789fda84211b6bd437 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260430 07:59:43.087831 9934 raft_consensus.cc:1275] T d3263a2b092d47c3b86f752e2bafcb63 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260430 07:59:43.088790 12369 leader_election.cc:290] T 58ea2e276db24da3890f352cef28ff97 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.090976 13142 leader_election.cc:290] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:43.091660 13142 raft_consensus.cc:2749] T f8946312605d455cbfe6546ad6873ec1 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260430 07:59:43.092916 13120 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.014899 13171 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:43.093967 13171 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:43.096431 13171 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:42.989012 13009 consensus_queue.cc:237] T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.098682 12920 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.099608 12798 leader_election.cc:290] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.101178 13157 raft_consensus.cc:2749] T 5cfb45b2ec3c45c38f2aba832ed29742 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260430 07:59:43.038482 12986 consensus_queue.cc:237] T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.103508 13143 leader_election.cc:290] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:43.037492 13099 leader_election.cc:290] T 1b60aa0987ed4a9185a3fe724e0acf23 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 election: Requested vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:43.030187 13113 leader_election.cc:290] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.106396 13111 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.109516 12661 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:42.983207 12970 raft_consensus.cc:515] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.018726 12858 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260430 07:59:43.034652 13105 consensus_queue.cc:260] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:43.121941 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:43.124650 10532 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:43.018687 13045 leader_election.cc:290] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:43.144150 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35278 (request call id 1683) took 2792 ms. Trace:
I20260430 07:59:43.144873 9854 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37838 (request call id 1975) took 1303 ms. Trace:
I20260430 07:59:43.184038 9854 rpcz_store.cc:276] 0430 07:59:41.841741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:41.842031 (+ 290us) service_pool.cc:225] Handling call
0430 07:59:41.842394 (+ 363us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:42.335856 (+493462us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:43.144533 (+808677us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b540029e354. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:43.144784 (+ 251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":710072576}
I20260430 07:59:42.977239 13081 raft_consensus.cc:515] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:42.982115 13135 raft_consensus.cc:515] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.202237 9933 rpcz_store.cc:276] 0430 07:59:40.351529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.403063 (+ 51534us) service_pool.cc:225] Handling call
0430 07:59:40.403390 (+ 327us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:40.446581 (+ 43191us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:40.446611 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:40.446649 (+ 38us) raft_consensus.cc:1645] Marking committed up to 4
0430 07:59:40.446679 (+ 30us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:43.143843 (+2697164us) spinlock_profiling.cc:243] Waited 43 ms on lock 0x7b54003246d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:43.143901 (+ 58us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:43.144080 (+ 179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98809600}
I20260430 07:59:43.209190 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8946312605d455cbfe6546ad6873ec1" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:43.210005 9933 raft_consensus.cc:2468] T f8946312605d455cbfe6546ad6873ec1 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:43.211300 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:43.211872 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.212602 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:43.213809 9977 leader_election.cc:304] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:43.216121 9977 leader_election.cc:304] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:43.217113 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829ed212483c445ba0dd4ae328bde3c2" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.217998 9977 leader_election.cc:304] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:43.219633 9977 leader_election.cc:304] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:43.220082 9978 leader_election.cc:304] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:43.221199 9977 leader_election.cc:304] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:43.223464 9977 leader_election.cc:304] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
I20260430 07:59:43.225154 9977 leader_election.cc:304] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
W20260430 07:59:43.222496 9828 leader_election.cc:336] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147): Timed out: RequestConsensusVote RPC to 127.8.119.130:38147 timed out after 1.780s (SENT)
W20260430 07:59:43.228753 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:43.099634 12920 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:43.234350 9861 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1920) took 2363 ms. Trace:
I20260430 07:59:43.239596 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:43.240043 9861 rpcz_store.cc:276] 0430 07:59:40.871402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.871565 (+ 163us) service_pool.cc:225] Handling call
0430 07:59:40.886233 (+ 14668us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:41.102186 (+215953us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:43.233950 (+2131764us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b5400208d54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:43.234273 (+ 323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":363038592}
W20260430 07:59:43.176900 13162 log.cc:927] Time spent T 58ea2e276db24da3890f352cef28ff97 P d66db415f0f0402fa431349afbf59888: Append to log took a long time: real 0.471s user 0.002s sys 0.000s
I20260430 07:59:43.245159 9933 raft_consensus.cc:2393] T a81d1c3211f94be6a77c0c0902142e68 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 9: Already voted for candidate 77e19ab653f540339c1c613fad1451b5 in this term.
I20260430 07:59:43.175971 13077 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260430 07:59:43.234884 13079 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.007s sys 0.004s
I20260430 07:59:43.176187 12661 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:43.254418 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:43.254436 9863 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.183465 13157 raft_consensus.cc:493] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:43.257609 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:43.132901 13151 leader_election.cc:290] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.258623 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:43.233899 12798 raft_consensus.cc:2749] T cd7ed510277642ca8409dce1775947c1 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260430 07:59:42.955807 13136 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:43.236112 13113 raft_consensus.cc:2749] T 2ad545d8bf83435f87a2b1169a24878e P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260430 07:59:43.263242 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 281202us
W20260430 07:59:43.263162 9938 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.838s user 0.000s sys 0.019s
I20260430 07:59:43.276162 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:43.234086 13111 raft_consensus.cc:493] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:43.234208 13143 raft_consensus.cc:2749] T 76d5ae1122e343f28f49a7fdbf3db843 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260430 07:59:43.281826 9938 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.879s user 0.000s sys 0.020s
I20260430 07:59:43.283694 9933 raft_consensus.cc:2410] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260430 07:59:43.258176 13079 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.007s sys 0.005s
I20260430 07:59:43.255604 13105 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Advancing to term 10
W20260430 07:59:43.288038 9976 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.014s sys 0.011s
W20260430 07:59:43.264467 10564 consensus_meta.cc:287] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888: Time spent flushing consensus metadata: real 0.736s user 0.012s sys 0.003s
I20260430 07:59:43.291858 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00df97e04911453d95498aacc19c7685" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
W20260430 07:59:43.292253 9976 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s user 0.014s sys 0.011s
I20260430 07:59:43.292518 9933 raft_consensus.cc:3060] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:43.288479 9938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1262) took 2567 ms. Trace:
I20260430 07:59:43.275230 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a66ec69e8f2c493fa90c40e25719d298" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:43.256668 13077 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:43.296862 9938 rpcz_store.cc:276] 0430 07:59:40.720981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:41.199854 (+478873us) service_pool.cc:225] Handling call
0430 07:59:41.200143 (+ 289us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:41.200359 (+ 216us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:41.200680 (+ 321us) log.cc:844] Serialized 14 byte log entry
0430 07:59:42.402589 (+1201909us) log.cc:844] Serialized 14 byte log entry
0430 07:59:43.287693 (+885104us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:43.287817 (+ 124us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:43.287892 (+ 75us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:43.288147 (+ 255us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:43.288399 (+ 252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":711024,"prepare.queue_time_us":202540,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":5950,"raft.queue_time_us":899554,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":6410,"spinlock_wait_cycles":42260864,"thread_start_us":1391148,"threads_started":4,"wal-append.queue_time_us":276755}
I20260430 07:59:43.288321 9934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1281) took 1083 ms. Trace:
I20260430 07:59:43.301441 10566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e18e051a7225424ebb64aedb21bb97df" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:43.305162 13113 raft_consensus.cc:2804] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 07:59:43.305969 10559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:43.257555 13120 consensus_queue.cc:1048] T a66ec69e8f2c493fa90c40e25719d298 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260430 07:59:43.306960 10565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02a63423ae464eb7b415a38419524150" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:43.012382 13161 raft_consensus.cc:515] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.248703 12920 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.285059 13039 leader_election.cc:290] T 7c393abc9c604fc9aff3cedf85f68725 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.308645 12798 raft_consensus.cc:2804] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260430 07:59:43.342409 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 113403us
I20260430 07:59:43.344735 9933 raft_consensus.cc:2410] T 00df97e04911453d95498aacc19c7685 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260430 07:59:43.276458 13157 raft_consensus.cc:515] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.346141 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1328) took 1603 ms. Trace:
I20260430 07:59:43.265915 12661 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:43.347962 13136 leader_election.cc:290] T 805814cfaf2b43ab87c90b2b7b56a211 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.297487 10564 raft_consensus.cc:2468] T 7303f2c98e1b411f83314a2c2f146d4b P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 7.
I20260430 07:59:43.365620 10010 rpcz_store.cc:276] 0430 07:59:41.742663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:42.954674 (+1212011us) service_pool.cc:225] Handling call
0430 07:59:42.955117 (+ 443us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:42.955351 (+ 234us) raft_consensus.cc:1534] Early marking committed up to index 5
0430 07:59:43.194536 (+239185us) log.cc:844] Serialized 14 byte log entry
0430 07:59:43.345263 (+150727us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:43.345391 (+ 128us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:43.345507 (+ 116us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:43.345648 (+ 141us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:43.345922 (+ 274us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":361905,"threads_started":2}
I20260430 07:59:43.326424 12927 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.327530 13113 raft_consensus.cc:493] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:43.299291 13111 raft_consensus.cc:515] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.341116 9938 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3028e67ff7d948ee8accee91fbcb70c3" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:43.398013 9937 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1268) took 1253 ms. Trace:
I20260430 07:59:43.477669 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.330255 13077 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:43.404435 9936 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1693) took 1221 ms (client timeout 1578 ms). Trace:
W20260430 07:59:43.514623 9936 rpcz_store.cc:269] 0430 07:59:42.183504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:42.341364 (+157860us) service_pool.cc:225] Handling call
0430 07:59:43.404241 (+1062877us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b540027fbd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:43.404370 (+ 129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":740961024}
W20260430 07:59:43.517416 9858 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37824 (request call id 1925) took 2626 ms (client timeout 1668 ms). Trace:
W20260430 07:59:43.518029 9858 rpcz_store.cc:269] 0430 07:59:40.885703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.897745 (+ 12042us) service_pool.cc:225] Handling call
0430 07:59:43.477506 (+2579761us) spinlock_profiling.cc:243] Waited 913 us on lock 0x7b5800000f98. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb566023941 00007fb5661a86c1 00007fb5661a8a3e 000000000035a3c9 00007fb56e1e08e0 00007fb56e2a9784 00007fb56e2a97f1 00007fb56e257760 00007fb56e257044 00007fb5677c4834 00007fb5677c47c2
0430 07:59:43.511875 (+ 34369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2120960}
I20260430 07:59:43.402812 9930 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1257) took 2719 ms. Trace:
I20260430 07:59:43.415405 9855 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1931) took 2348 ms. Trace:
I20260430 07:59:43.519918 9930 rpcz_store.cc:276] 0430 07:59:40.683284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:40.714340 (+ 31056us) service_pool.cc:225] Handling call
0430 07:59:42.128215 (+1413875us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:42.128932 (+ 717us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:43.402553 (+1273621us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54003241d0. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb569282c8d 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:43.402742 (+ 189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3166489472}
I20260430 07:59:43.509917 9938 raft_consensus.cc:2468] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 11.
W20260430 07:59:43.415483 9827 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:43.393240 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:43.509850 9937 rpcz_store.cc:276] 0430 07:59:42.144805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:42.864661 (+719856us) service_pool.cc:225] Handling call
0430 07:59:42.937503 (+ 72842us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:43.151093 (+213590us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:43.151142 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0430 07:59:43.151205 (+ 63us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:43.151257 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:43.397659 (+246402us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b5400337cd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:43.397726 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:43.397930 (+ 204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":490418432}
I20260430 07:59:43.378008 10563 raft_consensus.cc:2393] T a66ec69e8f2c493fa90c40e25719d298 P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 48e81934d52a4e93be5171ceea193032 in current term 7: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
I20260430 07:59:43.519989 9855 rpcz_store.cc:276] 0430 07:59:41.066926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:41.067215 (+ 289us) service_pool.cc:225] Handling call
0430 07:59:41.067729 (+ 514us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:41.582055 (+514326us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:43.415001 (+1832946us) spinlock_profiling.cc:243] Waited 500 ms on lock 0x7b540010cf54. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:43.415317 (+ 316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1150831360}
W20260430 07:59:43.522349 9902 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260430 07:59:43.522711 10533 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.010s sys 0.027s
W20260430 07:59:43.527951 10533 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.010s sys 0.028s
W20260430 07:59:43.528330 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 576768us
I20260430 07:59:43.531577 9938 raft_consensus.cc:1240] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 48e81934d52a4e93be5171ceea193032 for earlier term 5. Current term is 6. Ops: []
I20260430 07:59:43.378782 9933 raft_consensus.cc:1275] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260430 07:59:43.521694 9903 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:43.533926 9933 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35268 (request call id 1282) took 1328 ms. Trace:
I20260430 07:59:43.534687 9933 rpcz_store.cc:276] 0430 07:59:42.205119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:43.377967 (+1172848us) service_pool.cc:225] Handling call
0430 07:59:43.378367 (+ 400us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:43.533335 (+154968us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:43.533806 (+ 471us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:43.354141 13143 raft_consensus.cc:2804] T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260430 07:59:43.543462 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 240369us
I20260430 07:59:43.405972 13186 raft_consensus.cc:3060] T 18a52542f8aa4aeca8e319d4f60e9699 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:43.561599 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c37b1d9d0b54e12b8ce6cd28f1fdb07" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:43.422894 12661 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:43.514554 13113 raft_consensus.cc:3060] T e18e051a7225424ebb64aedb21bb97df P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:43.524423 13081 leader_election.cc:290] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:43.603627 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 324173us
I20260430 07:59:43.362924 12798 raft_consensus.cc:493] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:43.403891 13137 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:43.536475 13160 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:43.638366 9828 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f97a330f374849be58a168318c1ec6; no voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:43.583901 13135 leader_election.cc:290] T 022dabb87c374f5c8ae73005ad2fdad6 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:43.401618 13176 raft_consensus.cc:2804] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260430 07:59:43.512897 12970 leader_election.cc:290] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:43.624711 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 71047.7us
W20260430 07:59:43.637380 9975 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:43.426826 13161 leader_election.cc:290] T 19c3ca044c3d4377b31c4d07c513415c P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
W20260430 07:59:43.625368 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 65291.6us
I20260430 07:59:43.457664 13149 leader_election.cc:290] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:43.422596 13041 leader_election.cc:290] T 2ad545d8bf83435f87a2b1169a24878e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:43.556201 10564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54778 (request call id 155) took 1596 ms. Trace:
I20260430 07:59:43.386758 13120 consensus_queue.cc:1048] T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.197s
I20260430 07:59:43.498672 13078 raft_consensus.cc:493] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:43.637377 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:43.421185 12920 leader_election.cc:290] T 43b558e289a64355972c2e915f633bb9 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:43.396559 13157 leader_election.cc:290] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:43.640646 9828 leader_election.cc:304] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters: d66db415f0f0402fa431349afbf59888
W20260430 07:59:43.645949 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 244736us
W20260430 07:59:43.638845 13037 consensus_meta.cc:287] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5: Time spent flushing consensus metadata: real 0.628s user 0.009s sys 0.005s
I20260430 07:59:43.650005 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.663115 9858 raft_consensus.cc:2468] T 202cb0ccf6a947e6bdcffc383288322c P 27f97a330f374849be58a168318c1ec6 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d66db415f0f0402fa431349afbf59888 in term 7.
I20260430 07:59:43.629069 13031 leader_election.cc:290] T 1caba957d05d44788146efd70334d6fe P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:43.653141 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "202cb0ccf6a947e6bdcffc383288322c" candidate_uuid: "d66db415f0f0402fa431349afbf59888" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:43.675221 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:43.676566 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5"
I20260430 07:59:43.636548 12798 raft_consensus.cc:3060] T db24b74e2fa44ae2beba675582424a3b P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Advancing to term 5
W20260430 07:59:43.639104 9978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 404413us
I20260430 07:59:43.657853 10010 raft_consensus.cc:2468] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 8.
I20260430 07:59:43.622664 12947 consensus_queue.cc:1048] T b978c5a9ab41402c806352f5ace14295 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260430 07:59:43.416502 13170 consensus_queue.cc:1059] T 5cfb45b2ec3c45c38f2aba832ed29742 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.266s
I20260430 07:59:43.690207 9933 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "77e19ab653f540339c1c613fad1451b5" is_pre_election: true
I20260430 07:59:43.641428 13081 raft_consensus.cc:2749] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260430 07:59:43.669873 9858 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95ce138277864f41ba2e21b8b8d4a730" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.671393 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.693394 9937 raft_consensus.cc:1275] T 022dabb87c374f5c8ae73005ad2fdad6 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260430 07:59:43.693593 9854 raft_consensus.cc:2468] T c420e10ec8014b4f8df55f542d9693f3 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e19ab653f540339c1c613fad1451b5 in term 9.
I20260430 07:59:43.642566 13176 raft_consensus.cc:493] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:43.642262 12661 raft_consensus.cc:493] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:43.640297 12959 consensus_queue.cc:1059] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260430 07:59:43.664503 9938 raft_consensus.cc:1275] T 3028e67ff7d948ee8accee91fbcb70c3 P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Refusing update from remote peer d66db415f0f0402fa431349afbf59888: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260430 07:59:43.689154 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eaaca7c934d147a396bbaee7e10d6496" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.695904 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "058d2a87aec64ba1b745c5bd593c00e2" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.697695 9933 raft_consensus.cc:1275] T 3d83257ff86a443fbf15a8130dd8106e P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260430 07:59:43.661197 13135 raft_consensus.cc:493] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:43.721597 9937 raft_consensus.cc:1275] T eaaca7c934d147a396bbaee7e10d6496 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260430 07:59:43.702960 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b60aa0987ed4a9185a3fe724e0acf23" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:43.708633 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.723259 10010 raft_consensus.cc:3060] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:43.673780 12920 raft_consensus.cc:493] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:43.721663 9933 raft_consensus.cc:1275] T 1aa1e40099b3420eaada73851145a79e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260430 07:59:43.722941 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d83257ff86a443fbf15a8130dd8106e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.725203 9903 leader_election.cc:304] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032
I20260430 07:59:43.654237 10564 rpcz_store.cc:276] 0430 07:59:41.922657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:41.922821 (+ 164us) service_pool.cc:225] Handling call
0430 07:59:43.519057 (+1596236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:43.726559 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "926617cd6f3149f6a70ccb92f4504659" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.726809 9904 leader_election.cc:304] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:43.726886 9938 raft_consensus.cc:1275] T 058d2a87aec64ba1b745c5bd593c00e2 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Refusing update from remote peer 27f97a330f374849be58a168318c1ec6: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260430 07:59:43.725243 9933 raft_consensus.cc:1240] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Rejecting Update request from peer 27f97a330f374849be58a168318c1ec6 for earlier term 5. Current term is 6. Ops: []
W20260430 07:59:43.699747 13062 consensus_meta.cc:287] T 3d83257ff86a443fbf15a8130dd8106e P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.695s user 0.012s sys 0.000s
I20260430 07:59:43.725299 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19c3ca044c3d4377b31c4d07c513415c" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.732734 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "805814cfaf2b43ab87c90b2b7b56a211" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.705673 13157 raft_consensus.cc:493] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:43.736330 9862 raft_consensus.cc:2468] T 805814cfaf2b43ab87c90b2b7b56a211 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 5.
I20260430 07:59:43.736425 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ad545d8bf83435f87a2b1169a24878e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.698310 13037 raft_consensus.cc:493] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
W20260430 07:59:43.708452 13190 log.cc:927] Time spent T cd7ed510277642ca8409dce1775947c1 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
W20260430 07:59:43.744261 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 380857us
I20260430 07:59:43.706053 13176 raft_consensus.cc:3060] T d2df038061ff495dbdb2eaa4ad73c76e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Advancing to term 6
W20260430 07:59:43.775564 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 99600.5us
I20260430 07:59:43.708701 13149 raft_consensus.cc:493] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:43.690867 13078 raft_consensus.cc:515] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:43.640420 13137 raft_consensus.cc:515] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.709671 12661 raft_consensus.cc:515] T e537f351c19b43fd8b0b0a89cf897487 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:43.777648 9977 leader_election.cc:304] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:43.938922 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 213729us
W20260430 07:59:43.829799 9829 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260430 07:59:43.952472 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c393abc9c604fc9aff3cedf85f68725" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:43.833851 9862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "022dabb87c374f5c8ae73005ad2fdad6" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:43.856386 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3263a2b092d47c3b86f752e2bafcb63" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:44.000860 9903 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:43.815613 9857 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43b558e289a64355972c2e915f633bb9" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:43.727253 13138 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.110s user 0.008s sys 0.041s
W20260430 07:59:44.009176 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 231577us
I20260430 07:59:44.010672 10007 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44026 (request call id 1302) took 3380 ms. Trace:
W20260430 07:59:44.011751 9904 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 284907us
I20260430 07:59:43.733330 13160 raft_consensus.cc:493] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:44.027642 13160 raft_consensus.cc:515] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:44.019132 13197 raft_consensus.cc:493] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:44.019654 13208 raft_consensus.cc:2749] T 829ed212483c445ba0dd4ae328bde3c2 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260430 07:59:43.762224 13093 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
W20260430 07:59:43.844415 12955 consensus_meta.cc:287] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.796s user 0.008s sys 0.008s
I20260430 07:59:44.031188 13197 raft_consensus.cc:515] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:43.831038 13193 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20260430 07:59:44.029882 13160 leader_election.cc:290] T 9031ab0cc4144643b99f584e93b8c75f P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:44.120201 13170 consensus_queue.cc:1048] T d3263a2b092d47c3b86f752e2bafcb63 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260430 07:59:44.120464 10534 leader_election.cc:304] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, d66db415f0f0402fa431349afbf59888; no voters:
W20260430 07:59:44.165424 10533 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.029s sys 0.044s
I20260430 07:59:44.121583 10568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9fe0ed01a03d4f93a1f8a9644c5497cf" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:43.742328 13031 raft_consensus.cc:3055] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Stepping down as leader of term 8
W20260430 07:59:44.166061 10533 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.502s user 0.029s sys 0.044s
I20260430 07:59:44.126950 13137 leader_election.cc:290] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:44.133083 13044 leader_election.cc:290] T 7f111c6c48d34a6fb38048804967ed93 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:44.133438 13078 leader_election.cc:290] T 2947a88351384118b009f71f871ab9c8 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
W20260430 07:59:44.133725 9828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 99871.8us
I20260430 07:59:43.761521 13157 raft_consensus.cc:515] T 805814cfaf2b43ab87c90b2b7b56a211 P 77e19ab653f540339c1c613fad1451b5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:44.136152 12986 raft_consensus.cc:493] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:44.119279 10563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2947a88351384118b009f71f871ab9c8" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:44.125046 13066 raft_consensus.cc:3060] T ac5d1b9983db48e198134c6ab5246fbb P 48e81934d52a4e93be5171ceea193032 [term 10 FOLLOWER]: Advancing to term 11
I20260430 07:59:44.149948 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75807667b9164e0ba12c5013baff3b81" candidate_uuid: "48e81934d52a4e93be5171ceea193032" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:44.023864 9977 leader_election.cc:304] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
W20260430 07:59:43.866961 13184 log.cc:927] Time spent T 452aa377ba5f47c3bdd4bf5d4e48059d P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.237s user 0.000s sys 0.001s
I20260430 07:59:43.785223 13037 raft_consensus.cc:515] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
W20260430 07:59:43.955698 13121 log.cc:927] Time spent T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 48e81934d52a4e93be5171ceea193032: Append to log took a long time: real 2.211s user 0.003s sys 0.000s
W20260430 07:59:43.955971 13118 log.cc:927] Time spent T 7c10b37fe2e54d7a951545147ac2e381 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 2.210s user 0.003s sys 0.000s
W20260430 07:59:43.886065 13200 log.cc:927] Time spent T 9001555606564c6081e62a65ffa52dc8 P 27f97a330f374849be58a168318c1ec6: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
W20260430 07:59:43.957207 13105 consensus_meta.cc:287] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 0.627s user 0.008s sys 0.004s
W20260430 07:59:43.779083 13074 consensus_meta.cc:287] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6: Time spent flushing consensus metadata: real 1.372s user 0.009s sys 0.007s
I20260430 07:59:44.161057 9856 raft_consensus.cc:2410] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260430 07:59:43.779740 13189 log.cc:927] Time spent T a7e202af141f43f7b88e4231eed0ba94 P 77e19ab653f540339c1c613fad1451b5: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20260430 07:59:44.126036 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ea2e276db24da3890f352cef28ff97" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
I20260430 07:59:43.740391 12970 raft_consensus.cc:493] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d66db415f0f0402fa431349afbf59888)
I20260430 07:59:43.734126 13135 raft_consensus.cc:515] T 9fe0ed01a03d4f93a1f8a9644c5497cf P 48e81934d52a4e93be5171ceea193032 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:43.825824 13041 raft_consensus.cc:2804] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260430 07:59:43.831243 12927 leader_election.cc:290] T 82b70cfe134341abb8df72ad2244032e P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 6 election: Requested vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205)
I20260430 07:59:43.825146 13111 leader_election.cc:290] T d3263a2b092d47c3b86f752e2bafcb63 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:44.269495 9860 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37824 (request call id 1942) took 1328 ms. Trace:
W20260430 07:59:44.172344 9903 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 171553us
I20260430 07:59:43.761915 12920 raft_consensus.cc:515] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
W20260430 07:59:44.107285 13138 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.409s user 0.008s sys 0.042s
I20260430 07:59:44.141182 13006 raft_consensus.cc:2804] T b978c5a9ab41402c806352f5ace14295 P 27f97a330f374849be58a168318c1ec6 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260430 07:59:43.823500 13151 raft_consensus.cc:2749] T 9c37b1d9d0b54e12b8ce6cd28f1fdb07 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260430 07:59:44.278668 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.131:39205, user_credentials={real_user=slave}} blocked reactor thread for 111788us
I20260430 07:59:44.014422 12743 raft_consensus.cc:493] T bddad6955cd64efd850fddc61341be7e P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:44.114995 13196 raft_consensus.cc:493] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:43.887569 13080 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:44.278892 9861 raft_consensus.cc:2468] T 75807667b9164e0ba12c5013baff3b81 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e81934d52a4e93be5171ceea193032 in term 8.
W20260430 07:59:44.284322 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 262211us
W20260430 07:59:44.285677 10534 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.129:36679, user_credentials={real_user=slave}} blocked reactor thread for 165293us
I20260430 07:59:44.289062 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82b70cfe134341abb8df72ad2244032e" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:43.795418 13001 consensus_queue.cc:1048] T 2947a88351384118b009f71f871ab9c8 P d66db415f0f0402fa431349afbf59888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.724s
I20260430 07:59:44.172281 13031 raft_consensus.cc:740] T a81d1c3211f94be6a77c0c0902142e68 P 27f97a330f374849be58a168318c1ec6 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:44.014861 13149 raft_consensus.cc:515] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 77e19ab653f540339c1c613fad1451b5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:43.874788 13182 raft_consensus.cc:493] T 981fd927b4114a73b86c1e232f121c7f P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:44.237749 13160 raft_consensus.cc:493] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 27f97a330f374849be58a168318c1ec6)
I20260430 07:59:44.296525 12975 raft_consensus.cc:2804] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260430 07:59:44.292989 13078 raft_consensus.cc:2749] T a81d1c3211f94be6a77c0c0902142e68 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260430 07:59:44.312255 12963 raft_consensus.cc:493] T 18a52542f8aa4aeca8e319d4f60e9699 P 27f97a330f374849be58a168318c1ec6 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:44.320406 13204 raft_consensus.cc:493] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:44.321372 13041 raft_consensus.cc:493] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 48e81934d52a4e93be5171ceea193032)
I20260430 07:59:44.293076 13079 consensus_queue.cc:1048] T 022dabb87c374f5c8ae73005ad2fdad6 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.030s
I20260430 07:59:44.336737 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18a52542f8aa4aeca8e319d4f60e9699" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6"
I20260430 07:59:44.336802 9977 leader_election.cc:304] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:44.324887 9860 rpcz_store.cc:276] 0430 07:59:42.941264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:43.008433 (+ 67169us) service_pool.cc:225] Handling call
0430 07:59:43.027281 (+ 18848us) raft_consensus.cc:1471] Updating replica for 1 ops
0430 07:59:43.027703 (+ 422us) raft_consensus.cc:1534] Early marking committed up to index 0
0430 07:59:43.027745 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0430 07:59:43.028355 (+ 610us) log.cc:844] Serialized 26 byte log entry
0430 07:59:43.213801 (+185446us) raft_consensus.cc:1645] Marking committed up to 5
0430 07:59:43.286852 (+ 73051us) log.cc:844] Serialized 14 byte log entry
0430 07:59:43.287064 (+ 212us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:43.287133 (+ 69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0430 07:59:44.269063 (+981930us) raft_consensus.cc:1685] finished
0430 07:59:44.269092 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0430 07:59:44.269331 (+ 239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31845,"prepare.queue_time_us":422652,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":152980,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":199,"thread_start_us":118841,"threads_started":2,"wal-append.queue_time_us":259329}
I20260430 07:59:44.345698 9978 leader_election.cc:304] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f97a330f374849be58a168318c1ec6, 48e81934d52a4e93be5171ceea193032; no voters:
I20260430 07:59:44.304932 12955 consensus_queue.cc:1048] T 3d83257ff86a443fbf15a8130dd8106e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
W20260430 07:59:44.356060 9932 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35278 (request call id 1698) took 2170 ms (client timeout 1942 ms). Trace:
I20260430 07:59:44.300365 12986 raft_consensus.cc:515] T 2e8dad8acea649479d1e927cd32599fe P 27f97a330f374849be58a168318c1ec6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:43.796485 13202 consensus_queue.cc:1048] T df6bdf20bc534fb2a8de1b1c2cbd9af9 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.546s
I20260430 07:59:44.358395 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53754 (request call id 511) took 2708 ms. Trace:
I20260430 07:59:44.400121 10008 rpcz_store.cc:276] 0430 07:59:41.649753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:42.901157 (+1251404us) service_pool.cc:225] Handling call
0430 07:59:42.901602 (+ 445us) raft_consensus.cc:1471] Updating replica for 0 ops
0430 07:59:43.019191 (+117589us) raft_consensus.cc:1697] Filling consensus response to leader.
0430 07:59:44.358171 (+1338980us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b54000f7dd4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692841e1 00007fb569282c1e 00007fb56e255ad9 00007fb5677bfd34 00007fb5677bfcc2 00007fb5677bfbf1 00007fb5677bfb6c 00007fb5677bee22 00007fb5675be0ac 00007fb5675bd4e6
0430 07:59:44.358341 (+ 170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":250789760}
I20260430 07:59:44.310535 13074 raft_consensus.cc:515] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:44.309733 13081 consensus_queue.cc:1048] T 1aa1e40099b3420eaada73851145a79e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260430 07:59:44.381019 9854 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a81d1c3211f94be6a77c0c0902142e68" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
W20260430 07:59:44.383649 9977 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 46909us
W20260430 07:59:44.389760 9932 rpcz_store.cc:269] 0430 07:59:42.227879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:42.596942 (+369063us) service_pool.cc:225] Handling call
0430 07:59:44.355887 (+1758945us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54003377d4. stack: 00007fb5661b3c83 00007fb5661b3b80 00007fb565b01574 00007fb56f7d36b4 00007fb56f7d3649 00007fb5691ec951 00007fb5692876da 00007fb56e2571d7 00007fb5677c4834 00007fb5677c47c2 00007fb5677c46f1 00007fb5677c466c 00007fb5677c3922 00007fb5675be0ac 00007fb5675bd4e6 00007fb5675bce8f
0430 07:59:44.355998 (+ 111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1056744832}
I20260430 07:59:44.341415 12869 raft_consensus.cc:493] T ac5d1b9983db48e198134c6ab5246fbb P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:44.393081 9861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9031ab0cc4144643b99f584e93b8c75f" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "27f97a330f374849be58a168318c1ec6" is_pre_election: true
I20260430 07:59:44.411718 10008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c420e10ec8014b4f8df55f542d9693f3" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032" is_pre_election: true
I20260430 07:59:44.412750 9977 leader_election.cc:304] T 00df97e04911453d95498aacc19c7685 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48e81934d52a4e93be5171ceea193032; no voters: 27f97a330f374849be58a168318c1ec6, 77e19ab653f540339c1c613fad1451b5
I20260430 07:59:44.396853 13197 leader_election.cc:290] T e51196f9c4fb44bf8f6296f5394417bb P 27f97a330f374849be58a168318c1ec6 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:44.380716 10564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f111c6c48d34a6fb38048804967ed93" candidate_uuid: "77e19ab653f540339c1c613fad1451b5" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d66db415f0f0402fa431349afbf59888" is_pre_election: true
W20260430 07:59:44.403806 10010 consensus_meta.cc:287] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032: Time spent flushing consensus metadata: real 0.677s user 0.006s sys 0.005s
I20260430 07:59:44.418502 10010 raft_consensus.cc:2468] T 1b60aa0987ed4a9185a3fe724e0acf23 P 48e81934d52a4e93be5171ceea193032 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f97a330f374849be58a168318c1ec6 in term 9.
I20260430 07:59:44.419055 10010 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44026 (request call id 1345) took 1064 ms. Trace:
I20260430 07:59:44.419427 10010 rpcz_store.cc:276] 0430 07:59:43.354425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0430 07:59:43.683262 (+328837us) service_pool.cc:225] Handling call
0430 07:59:44.418994 (+735732us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260430 07:59:44.420750 10010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15f2cd3a47984929840836a88ecfd8b0" candidate_uuid: "27f97a330f374849be58a168318c1ec6" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "48e81934d52a4e93be5171ceea193032"
I20260430 07:59:44.386788 13208 raft_consensus.cc:2749] T eaaca7c934d147a396bbaee7e10d6496 P 48e81934d52a4e93be5171ceea193032 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260430 07:59:44.424903 13206 consensus_queue.cc:1048] T 058d2a87aec64ba1b745c5bd593c00e2 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260430 07:59:44.356850 13196 raft_consensus.cc:515] T 82b70cfe134341abb8df72ad2244032e P 48e81934d52a4e93be5171ceea193032 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:44.378589 13224 raft_consensus.cc:493] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:44.395426 13204 raft_consensus.cc:515] T 00df97e04911453d95498aacc19c7685 P 27f97a330f374849be58a168318c1ec6 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:44.414170 10008 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44010 (request call id 1945) took 1164 ms. Trace:
I20260430 07:59:44.378015 12975 raft_consensus.cc:493] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:44.382403 13041 raft_consensus.cc:3060] T c420e10ec8014b4f8df55f542d9693f3 P 77e19ab653f540339c1c613fad1451b5 [term 9 FOLLOWER]: Advancing to term 10
I20260430 07:59:44.324988 12970 raft_consensus.cc:515] T f8946312605d455cbfe6546ad6873ec1 P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } attrs { replace: true } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } } peers { permanent_uuid: "d66db415f0f0402fa431349afbf59888" member_type: VOTER last_known_addr { host: "127.8.119.132" port: 40723 } attrs { promote: false } }
I20260430 07:59:44.422652 9977 leader_election.cc:304] T 95ce138277864f41ba2e21b8b8d4a730 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters: 27f97a330f374849be58a168318c1ec6
I20260430 07:59:44.436184 9977 leader_election.cc:304] T 3028e67ff7d948ee8accee91fbcb70c3 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e81934d52a4e93be5171ceea193032, 77e19ab653f540339c1c613fad1451b5; no voters:
I20260430 07:59:44.423540 10010 raft_consensus.cc:3060] T 15f2cd3a47984929840836a88ecfd8b0 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:44.432922 9858 raft_consensus.cc:2393] T 95ce138277864f41ba2e21b8b8d4a730 P 27f97a330f374849be58a168318c1ec6 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77e19ab653f540339c1c613fad1451b5 in current term 9: Already voted for candidate 27f97a330f374849be58a168318c1ec6 in this term.
W20260430 07:59:44.438006 10533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.119.130:38147, user_credentials={real_user=slave}} blocked reactor thread for 61898.3us
I20260430 07:59:44.382520 13160 raft_consensus.cc:515] T 76d5ae1122e343f28f49a7fdbf3db843 P 77e19ab653f540339c1c613fad1451b5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:44.424521 13123 consensus_queue.cc:1048] T 43b558e289a64355972c2e915f633bb9 P 27f97a330f374849be58a168318c1ec6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.144s
I20260430 07:59:44.357925 13170 consensus_queue.cc:1059] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260430 07:59:44.398046 13111 raft_consensus.cc:2804] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260430 07:59:44.426103 13224 raft_consensus.cc:515] T 43b558e289a64355972c2e915f633bb9 P 77e19ab653f540339c1c613fad1451b5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:44.466300 13170 raft_consensus.cc:3055] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Stepping down as leader of term 5
I20260430 07:59:44.468520 13170 raft_consensus.cc:740] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f97a330f374849be58a168318c1ec6, State: Running, Role: LEADER
I20260430 07:59:44.438015 12975 raft_consensus.cc:3060] T 202cb0ccf6a947e6bdcffc383288322c P d66db415f0f0402fa431349afbf59888 [term 7 FOLLOWER]: Advancing to term 8
I20260430 07:59:44.469754 13170 consensus_queue.cc:260] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129" port: 36679 } } peers { permanent_uuid: "77e19ab653f540339c1c613fad1451b5" member_type: VOTER last_known_addr { host: "127.8.119.130" port: 38147 } } peers { permanent_uuid: "48e81934d52a4e93be5171ceea193032" member_type: VOTER last_known_addr { host: "127.8.119.131" port: 39205 } }
I20260430 07:59:44.472093 13170 raft_consensus.cc:3060] T 82b70cfe134341abb8df72ad2244032e P 27f97a330f374849be58a168318c1ec6 [term 5 FOLLOWER]: Advancing to term 6
I20260430 07:59:44.466952 13111 raft_consensus.cc:493] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 77e19ab653f540339c1c613fad1451b5)
I20260430 07:59:44.474947 12920 leader_election.cc:290] T 926617cd6f3149f6a70ccb92f4504659 P 48e81934d52a4e93be5171ceea193032 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 27f97a330f374849be58a168318c1ec6 (127.8.119.129:36679), 77e19ab653f540339c1c613fad1451b5 (127.8.119.130:38147)
I20260430 07:59:44.475215 13111 raft_consensus.cc:3060] T 75807667b9164e0ba12c5013baff3b81 P 48e81934d52a4e93be5171ceea193032 [term 8 FOLLOWER]: Advancing to term 9
I20260430 07:59:44.475706 13037 leader_election.cc:290] T 02a63423ae464eb7b415a38419524150 P 77e19ab653f540339c1c613fad1451b5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 48e81934d52a4e93be5171ceea193032 (127.8.119.131:39205), d66db415f0f0402fa431349afbf59888 (127.8.119.132:40723)
I20260430 07:59:44.476253 13037 raft_consensus.cc:493] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260430 07:59:44.476676 13037 raft_consensus.cc:515] T 7c10b37fe2e54d7a951545147ac2e381 P 77e19ab653f540339c1c613fad1451b5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27f97a330f374849be58a168318c1ec6" member_type: VOTER last_known_addr { host: "127.8.119.129"